- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper proposes Curvature-Harmonized Metric Descent (CHIME), a multi-objective optimization method for PINNs that jointly addresses gradient imbalance, directional conflicts, and ill-conditioned optimization geometry. CHIME formulates direction selection as a max–min optimization problem under a shared curvature metric, providing a first-order non-conflict guarantee and a geometric interpretation through the curvature-harmonized hull. The authors further establish a local residual-mode contraction bound and develop a practical SOAP-style implementation. Extensive experiments on PINN and multi-task learning benchmarks demonstrate competitive performance against existing objective-aware and curvature-aware optimization methods.
Critical strengths
- Simultaneously considering and balancing the effects of curvature and gradient conflicts in first-order PINN optimizers is indeed a novel perspective.
- The experimental evaluation is comprehensive, covering multiple PINN benchmarks, a wide range of competitive baselines, and additional multi-task learning tasks. CHIME demonstrates substantial improvements on several challenging problems, while the supplementary analyses further support its effectiveness.
Critical weaknesses
- Prior studies suggest that gradient conflicts and loss landscape pathology are not two entirely orthometric failure modes. Wang et al. (2025) [1] demonstrated that ill-conditioned curvature in the loss landscape can affect the alignment of gradient directions, while Rathore et al. (2024) [2] showed that PINN optimization difficulties largely stem from the numerical stiffness induced by PDE residual. These studies indicate that the two are largely coupled. Although Theorem 1 of CHIME relates worst-objective progress to the effective condition number of the joint residual kernel, it does not sufficiently explain the interaction between these two factors or why a shared curvature metric can improve both simultaneously. These leave the motivation for treating gradient conflicts and loss landscape pathology as two independent problems in this paper insufficiently clear. Therefore, I suggest that the authors strengthen the causal analysis regarding the motivation behind the method design. Additionally, Section 3 could be streamlined by reducing the presentation of established concepts from multi-objective and second-order optimization, leaving more room for the core analysis.
- The theoretical novelty of CHIME relative to existing variable-metric multi-objective optimization methods could be more clearly articulated. In particular, I suggest that the authors explicitly list two or three specific contributions in the Introduction. The current presentation (Lines 093–104) does not sufficiently clarify what novel improvements CHIME offers.
- The presentation of this manuscript is overly mathematical and lacks sufficient intuitive explanations. In particular, Section 4 introduces numerous abstract geometric concepts and dense mathematical formulations without adequately explaining the underlying motivation and intuition behind the proposed method. Given that this is primarily a practical optimization method rather than a purely theoretical study, I encourage the authors to prioritize accessibility for a broader machine learning audience and provide clearer explanations or illustrative example, such as Wang et al. (2025) or Rathore et al., (2024). More visualizations also help to further illustrate the proposed mechanism.
[1] S. Wang, A. K. Bhartari, B. Li, and P. Perdikaris, “Gradient Alignment in Physics-Informed Neural Networks: A Second-Order Optimization Perspective,” in NeurIPS, 2025.
[2] P. Rathore, W. Lei, Z. Frangella, L. Lu, and M. Udell, “Challenges in Training PINNs: A Loss Landscape Perspective,” in ICML, 2024.
Questions for authors
What are the key theoretical advances of CHIME over existing variable-metric multi-objective optimization methods? In particular, could the authors clarify which aspects of the proposed formulation and theoretical guarantees are fundamentally new, rather than extensions of existing shared-metric optimization frameworks?
4 comments
W2 and Q. Specific advances over variable-metric MOO
Residual-relative formulation and geometry. Sec. 3.3 identifies Chen et al. as the shared-metric optimization foundation. For positive residual losses, Sec. 4.1 uses
so the criterion maximizes the worst relative residual-norm reduction. App. B.4, Eqs. (68)–(69), establishes its exact connection to the shared-metric subproblem applied to these transformed objectives. The PINN-specific geometry is characterized by the loss-weighted curvature-harmonized hull in Definition 1 and Proposition 2; App. B.4, Eqs. (70)–(75), proves that its Euclidean conic hull recovers HARMONIC’s harmonized cone.
Objective-wise analysis beyond the generic shared-metric subproblem. To address the theoretical part of W1, we rewrote Sec. 4.2. Objective-Wise Progress and Residual Dynamics and Theorem 1. For each positive-weight objective in the fixed local linearized model, it gives the exact identity
where is the relative squared residual and the second term measures deviation orthogonal to its common radial motion. QP optimality establishes the common first-order progress , while Eq. (22) bounds the dispersion through the preconditioned kernel’s effective condition number and nullspace residual fraction. Both and this kernel depend on the same metric and selected coefficients. When , dispersion vanishes and the supported objectives share the factor .
App. B.7. Shared Whitening and Relative Residual Velocities (Theorem 2), then connects this result to the metric-construction target. Under the stated common-representability and inverse-GN approximation assumptions with positive metric-construction weights, it bounds the lagged-weight kernel’s conditioning and the objective-wise relative-velocity deviations. Exact shared whitening yields and for every objective, including those outside the current QP support. This explains how one shared metric can improve conditioning and harmonize relative residual dynamics. These residual-level conclusions are the additional theoretical contribution, beyond the shared-metric direction-selection principle.
W3. Intuition and visualization
We added Fig. 2 and integrated its panels into the explanation. The Sec. 4 uses the objective contours to introduce the stiff valley. Sec. 4.1, immediately after Eq. (14), explains the common ellipsoidal budget and how it restricts stiff directions while permitting larger movement along weakly curved directions. Sec. 4.3 connects the trajectory panel to the complementary roles of shared-metric estimation and current objective selection.
Thank you again for the thoughtful guidance on how to communicate the work. Your suggestions have been particularly valuable in connecting the motivation, mathematical contributions, empirical mechanisms, and visual explanation, making the revised paper easier to follow and its central contribution clearer to a broader audience.
I thank the authors for their clarifications. My main concerns were about novelty and clarity of writing, and I believe the revisions have largely addressed them. I have raised my score.
We are truly delighted to hear that the revisions have largely addressed your concerns. Your feedback has been invaluable in helping us strengthen the manuscript.
It is especially encouraging that these improvements have led you to reconsider your assessment. We are deeply grateful for your constructive guidance and willingness to revisit our work.
We thank the reviewer for recognizing the value of jointly considering curvature and gradient conflict, and for the constructive suggestions on theoretical positioning and accessibility. Your feedback helped us clarify why these mechanisms should be coupled, strengthen the supporting analysis and experiments, and explain the method through a visual guide and an explicit contribution statement.
W1. Coupling gradient conflict and ill-conditioning
Our distinction concerns what existing update rules explicitly control. Wang et al. (2025) analyze both intra-step and inter-step alignment and show the benefits of curvature-based optimization; Rathore et al. (2024) connect PDE-induced ill-conditioning to the PINN loss landscape and develop second-order optimization of the composite loss. CHIME builds on these insights by selecting objective combinations within the curvature geometry itself.
The revised Sec. 1. and Sec. 2. Table 1, make the algorithmic distinction explicit. DCGD, ConFIG, and HARMONIC construct first-order non-conflicting directions before subsequent optimizer transformations. SOAP$^\star$ and Muon$^\star$ combine curvature-aware updates with gradient-norm balancing, but improved alignment does not itself guarantee descent for every objective. In Sec. 4.1, CHIME uses the same preconditioner P in both C=G⊤PG and d⋆=PGα⋆. Thus, the coefficients account explicitly for how preconditioning changes objective-wise progress.
The new training-level controls test this coupling. In App. C.11. Component and Shared-Metric Ablations. (Table 19), the Poisson2d-C reference error is 0.0029, compared with 1.0000 for uniform coefficients, 0.8447 for an identity metric in both selection and update, and 0.6340 without residual normalization.
More directly, App. C.13. Poisson2d-C Training Dynamics and Metric Construction. (Table 27), replaces conflicting SOAP$^\star$/Muon$^\star$ proposals using a residual-normalized QP. Changing its metric from the native metric to the CHIME-style shared metric reduces the mean error from 0.6236 to 0.0096 for SOAP$^\star$ and from 0.3619 to 0.0122 for Muon$^\star$, over five matched repetitions of 50,000 steps at η=10−4. The SOAP comparison changes metric construction within the same optimizer family; the Muon comparison also changes the metric family. Importantly, both Muon QP controls have zero applied-update conflicts, yet markedly different errors. Eliminating conflicts alone therefore does not explain the accuracy gain; the geometry in which common descent is constructed also matters.
The revised theory below explains this interaction at the residual level. We also condensed the standard GN/Newton material in Secs. 3.2–3.3, retaining the definitions needed to connect first-order MOO, curvature preconditioning, and shared-metric optimization to CHIME.