acceptodds
Under review as a conference paper at ICLR 2027

Curvature-Harmonized Metric Descent for Multi-Loss Optimization in PINNs

Abstract

Training physics-informed neural networks (PINNs) requires jointly reducing multiple physics-based objectives despite gradient imbalance, directional conflicts, and ill-conditioned parameter-space geometry. Objective-aware methods can remain slow along ill-conditioned directions, whereas curvature-aware methods alone do not guarantee simultaneous descent across objectives. We propose **C**urvature-**H**armon**I**zed **ME**tric Descent (**CHIME**), which selects multi-objective directions in a shared curvature metric. **CHIME** maximizes the worst relative first-order reduction in residual norm under a step budget induced by this metric. The resulting direction lies in a curvature-harmonized hull, coupling a first-order non-conflict guarantee across objectives with curvature-aware preconditioning. A local linearized analysis shows that worst-objective progress and residual conditioning jointly control contraction of positive-eigenvalue residual modes. A practical SOAP-style realization estimates the shared metric and performs direction selection efficiently in objective space. Across four forward PINN benchmarks, **CHIME** reduces error by geometric mean factors of and relative to the best objective-aware and curvature-aware baseline for each benchmark, respectively. On the multi-task learning benchmark, it also achieves the best aggregate performance among the compared methods.

Primary area:
optimization
Keywords:
Physics-Informed Neural Networks, Multi-Objective Optimization, Gradient Conflict, Preconditioned Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3pSeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes Curvature-Harmonized Metric Descent (CHIME), a multi-objective optimization method for PINNs that jointly addresses gradient imbalance, directional conflicts, and ill-conditioned optimization geometry. CHIME formulates direction selection as a max–min optimization problem under a shared curvature metric, providing a first-order non-conflict guarantee and a geometric interpretation through the curvature-harmonized hull. The authors further establish a local residual-mode contraction bound and develop a practical SOAP-style implementation. Extensive experiments on PINN and multi-task learning benchmarks demonstrate competitive performance against existing objective-aware and curvature-aware optimization methods.

Critical strengths
  1. Simultaneously considering and balancing the effects of curvature and gradient conflicts in first-order PINN optimizers is indeed a novel perspective.
  2. The experimental evaluation is comprehensive, covering multiple PINN benchmarks, a wide range of competitive baselines, and additional multi-task learning tasks. CHIME demonstrates substantial improvements on several challenging problems, while the supplementary analyses further support its effectiveness.
Critical weaknesses
  1. Prior studies suggest that gradient conflicts and loss landscape pathology are not two entirely orthometric failure modes. Wang et al. (2025) [1] demonstrated that ill-conditioned curvature in the loss landscape can affect the alignment of gradient directions, while Rathore et al. (2024) [2] showed that PINN optimization difficulties largely stem from the numerical stiffness induced by PDE residual. These studies indicate that the two are largely coupled. Although Theorem 1 of CHIME relates worst-objective progress to the effective condition number of the joint residual kernel, it does not sufficiently explain the interaction between these two factors or why a shared curvature metric can improve both simultaneously. These leave the motivation for treating gradient conflicts and loss landscape pathology as two independent problems in this paper insufficiently clear. Therefore, I suggest that the authors strengthen the causal analysis regarding the motivation behind the method design. Additionally, Section 3 could be streamlined by reducing the presentation of established concepts from multi-objective and second-order optimization, leaving more room for the core analysis.
  2. The theoretical novelty of CHIME relative to existing variable-metric multi-objective optimization methods could be more clearly articulated. In particular, I suggest that the authors explicitly list two or three specific contributions in the Introduction. The current presentation (Lines 093–104) does not sufficiently clarify what novel improvements CHIME offers.
  3. The presentation of this manuscript is overly mathematical and lacks sufficient intuitive explanations. In particular, Section 4 introduces numerous abstract geometric concepts and dense mathematical formulations without adequately explaining the underlying motivation and intuition behind the proposed method. Given that this is primarily a practical optimization method rather than a purely theoretical study, I encourage the authors to prioritize accessibility for a broader machine learning audience and provide clearer explanations or illustrative example, such as Wang et al. (2025) or Rathore et al., (2024). More visualizations also help to further illustrate the proposed mechanism.

[1] S. Wang, A. K. Bhartari, B. Li, and P. Perdikaris, “Gradient Alignment in Physics-Informed Neural Networks: A Second-Order Optimization Perspective,” in NeurIPS, 2025.

[2] P. Rathore, W. Lei, Z. Frangella, L. Lu, and M. Udell, “Challenges in Training PINNs: A Loss Landscape Perspective,” in ICML, 2024.

Questions for authors

What are the key theoretical advances of CHIME over existing variable-metric multi-objective optimization methods? In particular, could the authors clarify which aspects of the proposed formulation and theoretical guarantees are fundamentally new, rather than extensions of existing shared-metric optimization frameworks?

4 comments

We thank the reviewer for recognizing the value of jointly considering curvature and gradient conflict, and for the constructive suggestions on theoretical positioning and accessibility. Your feedback helped us clarify why these mechanisms should be coupled, strengthen the supporting analysis and experiments, and explain the method through a visual guide and an explicit contribution statement.

W1. Coupling gradient conflict and ill-conditioning

Our distinction concerns what existing update rules explicitly control. Wang et al. (2025) analyze both intra-step and inter-step alignment and show the benefits of curvature-based optimization; Rathore et al. (2024) connect PDE-induced ill-conditioning to the PINN loss landscape and develop second-order optimization of the composite loss. CHIME builds on these insights by selecting objective combinations within the curvature geometry itself.

The revised Sec. 1. and Sec. 2. Table 1, make the algorithmic distinction explicit. DCGD, ConFIG, and HARMONIC construct first-order non-conflicting directions before subsequent optimizer transformations. SOAP$^\star$ and Muon$^\star$ combine curvature-aware updates with gradient-norm balancing, but improved alignment does not itself guarantee descent for every objective. In Sec. 4.1, CHIME uses the same preconditioner P\mathbf P in both C=G~⊤PG~\mathbf C=\widetilde{\mathbf G}^{\top}\mathbf P\widetilde{\mathbf G} and d⋆=PG~α⋆\mathbf d^\star=\mathbf P\widetilde{\mathbf G}\boldsymbol\alpha^\star. Thus, the coefficients account explicitly for how preconditioning changes objective-wise progress.

The new training-level controls test this coupling. In App. C.11. Component and Shared-Metric Ablations. (Table 19), the Poisson2d-C reference error is 0.00290.0029, compared with 1.00001.0000 for uniform coefficients, 0.84470.8447 for an identity metric in both selection and update, and 0.63400.6340 without residual normalization.

More directly, App. C.13. Poisson2d-C Training Dynamics and Metric Construction. (Table 27), replaces conflicting SOAP$^\star$/Muon$^\star$ proposals using a residual-normalized QP. Changing its metric from the native metric to the CHIME-style shared metric reduces the mean error from 0.62360.6236 to 0.00960.0096 for SOAP$^\star$ and from 0.36190.3619 to 0.01220.0122 for Muon$^\star$, over five matched repetitions of 50,000 steps at η=10−4\eta=10^{-4}. The SOAP comparison changes metric construction within the same optimizer family; the Muon comparison also changes the metric family. Importantly, both Muon QP controls have zero applied-update conflicts, yet markedly different errors. Eliminating conflicts alone therefore does not explain the accuracy gain; the geometry in which common descent is constructed also matters.

The revised theory below explains this interaction at the residual level. We also condensed the standard GN/Newton material in Secs. 3.2–3.3, retaining the definitions needed to connect first-order MOO, curvature preconditioning, and shared-metric optimization to CHIME.

W2 and Q. Specific advances over variable-metric MOO

Residual-relative formulation and geometry. Sec. 3.3 identifies Chen et al. as the shared-metric optimization foundation. For positive residual losses, Sec. 4.1 uses

g~i=gi2Li=∇(1/2logLi),\widetilde{\mathbf g}_i=\frac{\mathbf{g_i}}{2 L_i} = \nabla (1/2 log L_i),

so the criterion maximizes the worst relative residual-norm reduction. App. B.4, Eqs. (68)–(69), establishes its exact connection to the shared-metric subproblem applied to these transformed objectives. The PINN-specific geometry is characterized by the loss-weighted curvature-harmonized hull in Definition 1 and Proposition 2; App. B.4, Eqs. (70)–(75), proves that its Euclidean conic hull recovers HARMONIC’s harmonized cone.

Objective-wise analysis beyond the generic shared-metric subproblem. To address the theoretical part of W1, we rewrote Sec. 4.2. Objective-Wise Progress and Residual Dynamics and Theorem 1. For each positive-weight objective in the fixed local linearized model, it gives the exact identity

Ri(η)=(1−ηq)2+η2αi⋆∥[δα]i∥22,R_i(\eta)=(1-\eta q)^2+\frac{\eta^2}{\alpha_i^\star}\|[\boldsymbol\delta_{\boldsymbol\alpha}]_i\|_2^2,

where RiR_i is the relative squared residual and the second term measures deviation orthogonal to its common radial motion. QP optimality establishes the common first-order progress qq, while Eq. (22) bounds the dispersion through the preconditioned kernel’s effective condition number and nullspace residual fraction. Both qq and this kernel depend on the same metric and selected coefficients. When κeff=ζ=1\kappa_{\mathrm{eff}}=\zeta=1, dispersion vanishes and the supported objectives share the factor (1−ηq)2(1-\eta q)^2.

App. B.7. Shared Whitening and Relative Residual Velocities (Theorem 2), then connects this result to the metric-construction target. Under the stated common-representability and inverse-GN approximation assumptions with positive metric-construction weights, it bounds the lagged-weight kernel’s conditioning and the objective-wise relative-velocity deviations. Exact shared whitening yields Jid⋆=cri\mathbf J_i\mathbf d^\star=c\mathbf r_i and Ri(η)=(1−ηc)2R_i(\eta)=(1-\eta c)^2 for every objective, including those outside the current QP support. This explains how one shared metric can improve conditioning and harmonize relative residual dynamics. These residual-level conclusions are the additional theoretical contribution, beyond the shared-metric direction-selection principle.

W3. Intuition and visualization

We added Fig. 2 and integrated its panels into the explanation. The Sec. 4 uses the objective contours to introduce the stiff valley. Sec. 4.1, immediately after Eq. (14), explains the common ellipsoidal budget and how it restricts stiff directions while permitting larger movement along weakly curved directions. Sec. 4.3 connects the trajectory panel to the complementary roles of shared-metric estimation and current objective selection.

Thank you again for the thoughtful guidance on how to communicate the work. Your suggestions have been particularly valuable in connecting the motivation, mathematical contributions, empirical mechanisms, and visual explanation, making the revised paper easier to follow and its central contribution clearer to a broader audience.

I thank the authors for their clarifications. My main concerns were about novelty and clarity of writing, and I believe the revisions have largely addressed them. I have raised my score.

We are truly delighted to hear that the revisions have largely addressed your concerns. Your feedback has been invaluable in helping us strengthen the manuscript.

It is especially encouraging that these improvements have led you to reconsider your assessment. We are deeply grateful for your constructive guidance and willingness to revisit our work.

Reviewer_S2BLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors introduce CHIME, a multi-objective optimization method designed for PINNs. CHIME transforms a shared-metric multi-objective optimization problem in parameter space into a quadratic programming problem in object-weight space. Compared to the standard shared-metric method, CHIME uses the normalized gradient and a SOAP-style metric as the shared metric. The resulting method achieves both curvature awareness and conflict-freeness during the optimization.

Critical strengths
  • CHIME is the first work to successfully cover both curvature awareness and conflict-freeness for PINN optimization. Although which part is more important remains an open question and depends on the specific PINN setup, the reviewer is happy to see research that addresses both points simultaneously.
Critical weaknesses
  • The motivation for each component in CHIME is less clear. CHIME seems to rely heavily on the shared-metric optimization method proposed by Chen et al. (Eq. 13), but the connection between CHIME and Chen et al.'s method is not well addressed. For example, the reason behind choosing SOAP to build a shared metric is unclear. Compared to other possible shared metrics, does SOAP bring any strong improvement?
  • This is a mathematically heavy paper, but less well organized. Many discussions in the method are good to include but not necessary for the main content. The current version of the manuscript makes it difficult for the audience to grasp its core idea. The reviewer would suggest focusing on the really important parts, e.g., sec 4.1 and sec 4.3, and reducing the other parts in Section 3 and Section 4.
Questions for authors
  • The reviewer is not very familiar with the work of Chen et al. (2023). According to the authors' discussion, it seems that the only constraint on the shared B\mathbf{B} is that it must be positive definite. Can the authors comment more on the effect of the choice of B\mathbf{B}? Why do the authors specifically choose the SOAP style of approximation? Is there any other practical design?
  • Although the authors did not explicitly consider the conflict-free property in the design of CHIME, the final method naturally satisfies it (Proposition 1). Can the authors comment on which step from Eq.13 to Eq.17 actually brings conflict-freeness to CHIME, or will a simple shared-metric multi-objective optimization suffice to make the update conflict-free?
4 comments

We thank the reviewer for recognizing the value of jointly addressing curvature and gradient conflict in PINNs. Your questions about the shared metric and suggestions on presentation helped us clarify the connection to prior optimization methods, evaluate alternative metric designs, and make the formulation and algorithm easier to follow.


W1. Connection to Chen et al. and motivation for the components

CHIME specializes shared-metric MOO to relative residual progress and provides a structured realization for PINN training. Sec. 3.3, Eq. (12), states the shared-metric formulation of Chen et al.; Sec. 4.1 then uses g~i=gi/(2Li)=∇(1/2log⁡Li)\widetilde{\mathbf g}_i = \mathbf g_i/(2L_i)=\nabla (1/2 \log L_i) to optimize relative residual reduction. App. B.4, Eqs. (68)–(69), establishes the precise connection: applying the shared-metric formulation to these normalized gradients gives the same direction as CHIME under the matched budget. Sec. 4.2 and App. B.7 further analyze the resulting objective-wise residual dynamics and the shared-whitening target.

The revised App. C.11, Component and Shared-Metric Ablations, also tests these choices through training. On Poisson2d-C, replacing QP weights with a uniform mixture, using an identity metric in both selection and update, or removing residual normalization changes the error from 0.00290.0029 to 1.00001.0000, 0.84470.8447, and 0.63400.6340, respectively (Table 19). Thus, the residual-relative criterion, objective weighting, and curvature preconditioning each have a concrete PINN training-level motivation. The alternative-metric comparisons addressing the SOAP choice are summarized under Q1 below.

Q1. Effect of the metric, the SOAP choice, and practical alternatives

Positive definiteness makes a metric admissible; its geometry and approximation quality determine its practical effect. Sec. 3.2 explains SOAP’s conditional layerwise inverse-GN interpretation. App. B.7, Shared Whitening and Relative Residual Velocities, identifies the corresponding target: under its common-representability and inverse-GN assumptions, exact shared whitening yields Jid⋆=cri\mathbf J_i\mathbf d^\star=c\mathbf r_i for every objective, giving a common relative residual rate.

Computationally, Sec. 4.3 and App. B.9 implement the metric using lagged weighted objective gradients, layerwise Gram statistics, and moments. App. B.9 gives the storage bound

O ⁣(∑ℓ[(dℓr)2+(dℓc)2+Dℓ]+mD+m2),O\!\left(\sum_\ell\big[(d_\ell^{\mathrm r})^2+(d_\ell^{\mathrm c})^2+D_\ell\big]+mD+m^2\right),

where mm is the number of objectives and Dℓ=dℓrdℓcD_\ell=d_\ell^{\mathrm r}d_\ell^{\mathrm c}. Coefficient selection requires only an mm-variable QP, with O(m2D)O(m^2D) Gram contractions after rotation. The measured peak allocated memory on Wave1d-C is 148.6148.6 MiB for CHIME and 148.7148.7 MiB for SOAP$^\star$ (App. D.3); CHIME takes 2.052.05 seconds per 100 steps on this problem (App. D.1).

Empirically, App. C.11 compares practical alternatives at η=10−4\eta=10^{-4} over 50,000 steps and five repetitions. The mean relative L2L^2 errors on Wave1d-C and Poisson2d-C are:

VariantWave1d-CPoisson2d-C
CHIME reference, SOAP-style shared metric0.00330.0029
Identity QP / SOAP update0.06680.0390
Diagonal metric (Adam)0.02610.0098
Polar-SPD metric0.52320.5821
Shampoo-style metric0.16040.7265

The identity-QP control changes coefficient selection while retaining the SOAP update; the other metric variants retain residual normalization and metric-consistent QP selection. These results support both the SOAP-style construction and selecting coefficients within the update metric. The formulation accommodates other positive-definite metrics, and Sec. 6 identifies more faithful and computationally efficient curvature approximations as a direction for further development.

W2. Organization and emphasis on the core method

We have condensed the standard GN/Newton discussion in Secs. 3.2–3.3 and added Fig. 2 as a geometric guide to the core formulation and implementation. Its metric-budget panel is explained in Sec. 4.1, while its trajectory panel is connected to metric estimation and direction selection in Sec. 4.3.

We also rewrote Sec. 4.2, Objective-Wise Progress and Residual Dynamics, around the objective-wise residual decomposition and its spectral-dispersion bound, with the detailed derivations and shared-whitening analysis in Apps. B.5–B.7. This keeps the main explanation centered on how the direction is selected and implemented, while retaining the analysis that explains the role of the metric.

Q2. Where conflict-freeness enters

Solving the shared-metric multi-objective subproblem to optimality already yields a non-conflicting selected direction. In CHIME, this property comes from maximizing common first-order progress in Eq. (14); the dual reformulation preserves it.

We now state the existence of strictly positive common progress directly in Sec. 4.1, immediately after Eq. (14). App. B.2. Positive Progress Before Pareto Stationarity. Lemma 1, proves that, for Li>0L_i>0, B≻0\mathbf B\succ0, and any budget q>0q>0, with P=B−1\mathbf P=\mathbf B^{-1} and C=G~⊤PG~\mathbf C=\widetilde{\mathbf G}^{\top}\mathbf P\widetilde{\mathbf G},

\qquad \gamma^\star=\sqrt{q\chi}, \qquad \gamma^\star>0\ \Longleftrightarrow\ \boldsymbol\theta\text{ is not Pareto-stationary}.$$ At Pareto stationarity, $\gamma^\star=0$. Away from stationarity, the matched budget $q=\chi$ gives $\mathbf d^\star=\mathbf P\widetilde{\mathbf G}\boldsymbol\alpha^\star$, and the QP optimality conditions imply $$\mathbf g_i^\top\mathbf d^\star =2L_i(\mathbf C\boldsymbol\alpha^\star)_i \ge 2L_iq>0 \qquad\text{for every }i.$$ This is the selected-direction guarantee in Proposition 1, proved in **App. B.3**. SOAP determines the curvature geometry, and residual normalization determines the relative-progress criterion; the non-conflict mechanism is the shared-metric common-progress optimization itself. --- Thank you again for the constructive review. Your questions helped us make the relationship to shared-metric MOO explicit, substantiate the practical metric choice, and connect the geometric explanation to the algorithm and its guarantees.

Thank you for your detailed responses and efforts in revising the manuscript. Your clarifications have helped me better understand the paper, and I believe the revisions have also improved its clarity and readability.

Considering that my initial assessment was already positive, I have decided to maintain my original score. While I find this work interesting, I would also like to acknowledge the limitations of my confidence in fully assessing the correctness of all technical details. Therefore, I would encourage the AC to consider the assessments of other reviewers alongside mine when making the final decision.

Thank you very much for your thoughtful follow-up. We are genuinely pleased to hear that our clarifications were helpful and that the revisions improved the manuscript's clarity and readability.

We sincerely appreciate your taking the time to revisit our responses and engage with our work throughout the review process. Thank you again for your constructive feedback and encouragement.

Reviewer_5PSAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Training Physics-Informed Neural Networks (PINNs) requires the simultaneous minimization of multiple loss components (e.g., partial differential equation (PDE) residuals, initial/boundary conditions, etc.). This paper notes that existing approaches have specific limitations: "Objective-aware" methods (such as DCGD, ConFIG, and HARMONIC) ensure that gradients from different loss components do not conflict, but they fail to account for curvature information. "Curvature-aware" methods (such as SOAP* and Muon*) employ preconditioning techniques but cannot guarantee that all loss components will decrease simultaneously.

The CHIME method integrates both aspects: Direction selection. Given a shared curvature metric BB, the method aims to maximize the "relative first-order residual decrease" for the worst-performing objective, subject to the step-size constraint ∥d∥B2≤q\|d\|^2_B \leq q (Equation 15). This is equivalent to solving a small quadratic programming (QP) problem (Equation 17): First, each gradient is normalized using 2Li2L_i. A Gram matrix CC is computed under the metric P=B−1P = B^{-1}. A combination of gradients is sought on a simplex such that the norm of this combination is minimized. Theoretical foundation. Proposition 1 states that the resulting direction has a non-negative inner product with each individual gradient. Proposition 2 establishes that this direction is, in fact, the metric projection of the origin onto the "curvature-coordinated convex hull." Theorem 1 provides a contraction bound—(1−ηq/κeff)(1 - \eta q/\kappa_{\text{eff}})—which applies to residual modes associated with positive eigenvalues ​​under local linearization. Implementation. The method utilizes a SOAP-style metric and aggregates gradient estimates via a weighted sum, where the weighting coefficients (α\alpha) are derived from the computation in the previous step. In the objective space, this requires solving only an m×mm \times m quadratic programming (QP) problem. Experiments. The study covers four forward PINN problems as well as the HInv inverse problem, with the experimental workflow following the standards established by the HARMONIC method. Multi-task learning experiments were conducted on the NYU-v2 and CelebA datasets. The appendix includes additional PINNacle benchmark problems, tables detailing computational costs, comparisons of wall-clock execution times, and an analysis of learning rates. Conclusion: For forward problems, CHIME outperforms the best-performing "objective-aware" baseline (with an average improvement of 5.85x) and the best-performing "curvature-aware" baseline (with an average improvement of 5.45x). On the HInv problem, however, it underperforms relative to DCGD, ConFIG, and HARMONIC. On the NYU-v2 dataset, it achieves the best overall performance metrics.

Critical strengths
  1. The problem is well-identified. The simultaneous occurrence of "imbalanced loss scales," "conflicting gradients," and "ill-conditioning" represents a genuine pain point for PINNs. Integrating gradient selection into the same metric used for parameter updates ensures logical consistency.
  2. The experimental data is comprehensive. The appendix provides 95% confidence intervals, problem-specific tables, results for two different learning rates, per-step execution times, and comparisons based on wall-clock time.
  3. The study includes long-duration training curves and mini-batch analyses. It also provides a detailed breakdown of how the geometric means in Table 2 were calculated.
  4. The scope is broad. Beyond the main text, the paper covers over 14 PINN problems and two multi-task learning (MTL) benchmarks. On the Poisson2d-C problem, CHIME achieves an error of approximately 0.003, whereas SOAP*, Muon*, and KFAC* yield values ​​around 0.66—a significant performance gap that demonstrates the method's effectiveness in certain scenarios.
  5. The gap between theory and practice is addressed. Appendix B.6 acknowledges that the use of momentum (β1>0\beta_1 > 0) means the guarantee of non-conflicting gradients holds only under additional conditions.
Critical weaknesses

W1. The sources of performance improvement are not disentangled. CHIME introduces four simultaneous changes—residual normalization, a quadratic programming (QP) problem for α\alpha based on a SOAP-derived metric, a metric estimated from the weighted aggregate of gradients from the previous step, and a SOAP preconditioner—yet no ablation studies are conducted at the training level. Necessary control experiments should include: SOAP combined with residual normalization using uniform weights or low-rank approximation (LRA) weights; using an identity matrix as the metric in the QP; combining HARMONIC/ConFIG directions with SOAP; and a version that excludes residual normalization. The only ablation study provided (Figure 9) examines the "first-order progress of the worst-case objective function"; since CHIME is explicitly designed to optimize this metric, this experiment merely validates Proposition 2 and fails to demonstrate actual gains in training performance.

W2. The theoretical foundation is weak, and the theoretical guarantees do not directly extend to the proposed algorithm.

  1. Propositions 1 and 2 represent classic properties of Multi-Objective Gradient Descent (MGDA)—a framework already explored by Chen et al. (2023) in the context of shared metrics—rendering claims regarding "innovation" overstated.
  2. The bound provided in Theorem 1 is even weaker than a trivial bound. According to Equations (81)–(82), q/κeff≤λmin⁡+q/\kappa_{\text{eff}} \le \lambda_{\min}^+; consequently, the term 1−ηq/κeff1 - \eta q/\kappa_{\text{eff}} is actually less effective than 1−ηλmin⁡+1 - \eta \lambda_{\min}^+ (derived from Equation (83)). This conclusion holds for any d=PJTed = PJ^T e and thus provides no specific insight into α\alpha or qq.
  3. The paper refers to the second-order moment used in SOAP as a "curvature metric"; however, this term is derived from the aggregated gradient rather than the JiTJiJ_i^T J_i matrix of any individual objective function. 4. When β1=0.9\beta_1 = 0.9, the update process is effectively a momentum step rather than an exact solution to the QP problem; consequently, the "conflict-free" guarantee holds only under an unverified condition. Although the paper does not report empirical conflict rates, it uses the term "conflict-free" in both the abstract and Table 1.

W3. Issues with core empirical results.

  1. The 5.45x performance improvement over baselines that incorporate curvature information is driven almost entirely by the Poisson2d-C experiment (where the improvement is approximately 227x, compared to values ​​around 0.66 for KFAC*, Muon*, and SOAP*). Excluding this experiment, the geometric mean improvement across the remaining three experiments is approximately 1.6x.
  2. In the HNd and Volterra1d experiments, the difference between the methods (0.0001 vs. 0.0002) appears only at the fourth decimal place, and their confidence intervals overlap.
  3. For the Wave1d-C experiment, when compared at the same wall-clock time (see Table 20), SOAP* achieves 0.0042 (±0.0010) and CHIME achieves 0.0035 (±0.0010)—a difference that falls within the margin of error.
  4. The computational cost per step for CHIME is 1.73–2.18 times that of SOAP*, and the paper provides no information regarding memory usage.

W4. Transparency regarding baselines and hyperparameter tuning.

  1. Based on Table 2 and the confidence intervals in Appendix E, it appears that CHIME uses a learning rate (lrlr) of 1e−41e-4 for the forward problems and 1e−31e-3 for the HInv problems, whereas the baseline methods utilize the optimal learning rate for each individual experimental setting. If this is indeed the case, it constitutes post-hoc selection based on test error—a point that is not disclosed in the main text. Please confirm whether this interpretation is accurate.
  2. In the Poisson2d-C experiment, the results for SOAP*, Muon*, and KFAC* all plateau around 0.66; however, the paper provides no analysis as to why this occurs.
  3. The paper does not evaluate ENGD and SNGD (listed in Table 1), Adam+L-BFGS, or NNCG (mentioned in related work); furthermore, the baseline combination of HARMONIC/ConFIG + SOAP is also not evaluated.
Questions for authors
  1. Add ablation studies: identity matrix initialization, uniform vs. LRA weighting, removal of residual normalization, and SOAP+HARMONIC.
  2. Include results for ENGD, SNGD, and L-BFGS (for at least one or two problems).
  3. Report the observed conflict rate.

For additional questions, see "Weaknesses."

4 comments

We sincerely thank the reviewer for the exceptionally detailed and thoughtful review. Your comments helped us identify several important gaps in the original manuscript, particularly in disentangling the contributions of individual components, clarifying the theoretical guarantees, and connecting the observed optimization behavior to the underlying physical problem. In response, we have substantially expanded the training-level ablations, theoretical analysis, and empirical diagnostics. We address each concern below.


W1 & Q1. Disentangling the contributions of individual components

We agree that the original first-order progress analysis alone was insufficient to establish the actual training benefits of CHIME. We have therefore added training-level ablations that separately examine objective selection, curvature preconditioning, and residual normalization (Appendix C.9 and C.11).

First, Table 19 compares CHIME against three component controls on Wave1d-C and Poisson2d-C. Replacing the QP-selected coefficients with a uniform mixture increases the Poisson2d-C error from 0.0029 to 1.0000. Replacing the shared metric with the identity increases it to 0.8447, while removing residual normalization increases it to 0.6340. The uniform-mixture variant uses equal weights in both the current update and subsequent metric estimation, while the identity-metric variant sets P=IP=I in both coefficient selection and the parameter update.

Second, to isolate metric-consistent coefficient selection from the parameter preconditioner, Table 20 introduces an identity-QP control that retains the SOAP update but selects coefficients using the Euclidean gradient Gram matrix. Its errors increase to 0.0668 on Wave1d-C and 0.0390 on Poisson2d-C, compared with 0.0033 and 0.0029 for CHIME. We additionally compare polar-SPD, Shampoo-style, and diagonal metrics. The SOAP-style metric achieves lower errors on Wave1d-C and Poisson2d-C than the alternative shared operators.

Table 21 further examines diagonal preconditioning and SOAP*/Muon* with LRA-style weights refreshed every step. These frequent-balancing controls retain substantial errors on Wave1d-C and Poisson2d-C, highlighting the difference between adaptive loss balancing and metric-consistent relative-progress selection.

Finally, Appendix C.12 includes HARMONIC+SOAP, which retains HARMONIC's objective-combination rule while applying SOAP to its selected direction. On Poisson2d-C, HARMONIC+SOAP achieves a mean error of 0.0044, compared with 0.0029 for CHIME.

Together, these experiments provide training-level evidence that the improvement comes from the interaction of residual-relative scaling, metric-consistent objective selection, and shared curvature preconditioning.

W2 & Q3. Theoretical contribution and conflict-free guarantees

We have revised the theoretical presentation to clarify the relationship with Chen et al. (2023) and strengthen the analysis of objective-wise residual dynamics (Sections 3.3 and 4.1–4.2; Appendix B.6–B.7).

We use the shared-metric multi-objective formulation of Chen et al. as the starting point. Our formulation introduces residual-relative gradients, g~i=gi/(2Li)=∇(12log⁡Li)\widetilde{\mathbf g}_i=\mathbf g_i/(2L_i)=\nabla (\frac12\log L_i), and selects their combination by maximizing the worst first-order relative residual progress under the shared curvature metric. This leads to the curvature-harmonized hull and its metric-projection characterization. Thus, the theoretical contribution lies in the residual-relative formulation, its induced geometry, and the resulting objective-wise residual analysis.

We have also revised Theorem 1. The revised theorem provides an exact decomposition:

Ri(η)=(1−ηq)2+η2αi⋆∥[δα]i∥22.R_i ( \eta) = (1 - \eta q)^{2} + \frac{\eta^2}{\alpha_i^\star} \|[\boldsymbol\delta_\alpha]_i\|_2^2.

Its accompanying bound explicitly characterizes the dispersion through the effective condition number κeff\kappa_{\mathrm{eff}} and the residual component outside the positive eigenspace, 1−ζ1-\zeta (Eqs. 21–22). This separates the common objective-wise relative progress from the spectral dispersion affecting individual residuals.

Furthermore, Appendix B.7 introduces Theorem 2, which characterizes shared inverse-GN whitening. In particular, it defines the weighted shared GN matrix

Bω=∑iωiJi⊤Ji∥ri∥22\mathbf B_\omega = \sum_i \omega_i \frac{\mathbf J_i^\top\mathbf J_i}{\|\mathbf r_i\|_2^2}

and establishes conditional bounds on the effective conditioning and objective-wise relative velocities when the shared preconditioner approximates its inverse. In the exact whitening limit, all objectives exhibit the same relative residual contraction. This provides a theoretical connection between the shared metric constructed from lagged weighted gradient aggregates and the relative residual dynamics.

Regarding the SOAP metric, we clarify in Appendix B.9 that its implementation maintains shared gradient statistics from the lagged weighted aggregate, including cross-objective terms. The shared inverse-GN analysis is presented as a conditional characterization of its curvature target.

Regarding conflict-freeness, positive definiteness of the shared metric alone is insufficient. The non-conflict property follows from the simplex-constrained QP optimality and its KKT conditions, as established in Proposition 1. The resulting guarantee applies to the instantaneous metric-selected direction before subsequent momentum transformations. We have made this scope explicit in the definition accompanying Table 1.

Importantly, we now also report empirical conflict rates in Appendix C.13. On Poisson2d-C, the selected CHIME direction has a mean of only 0.0005 conflicted objectives per step, whereas the actual applied update, after momentum, has a conflict rate of 42.62%. The corresponding rates are 90.81% for HARMONIC, 96.18% for ConFIG, 99.20% for Muon*, and 96.06% for SOAP*. Appendix B.9 additionally specifies the condition under which temporal momentum averaging preserves the non-conflict property.

W3. Empirical performance and computational cost

We appreciate the concern that the 5.45 geometric-mean improvement over curvature-aware baselines is strongly influenced by Poisson2d-C. Indeed, excluding Poisson2d-C, the improvement across the remaining three forward problems is approximately 1.6 using the means. We have added a detailed investigation of the Poisson2d-C performance gap in Appendix C.13.

The key diagnostic is the hole flux associated with the perforated domain. An independent reference calculation gives a mean flux of 5.5851. SOAP* and Muon* both produce approximately zero flux, whereas CHIME recovers 5.5844. HARMONIC and HARMONIC+SOAP also recover the reference flux. These results identify the plateau of SOAP* and Muon* with their failure to represent the required nonzero hole-flux component.

We further investigate the interaction between conflict resolution and metric construction through controlled interventions. For SOAP*, introducing a residual-normalized QP with its native metric improves the error to 0.6236, while using the shared metric reduces it further to 0.0096. For Muon*, the corresponding errors are 0.3619 and 0.0122. These results demonstrate the importance of shared metric construction beyond conflict correction alone.

We also acknowledge that the differences on HNd and Volterra1d are small. Their confidence intervals overlap, and we do not interpret these results as statistically separated performance improvements. Similarly, the wall-clock-matched Wave1d-C results are 0.0042 for SOAP* and 0.0035 for CHIME, with overlapping reported variability. These comparisons are now provided in Appendix E.1 and Appendix D.2.

Regarding computational cost, Appendix D.1 reports that CHIME requires approximately 1.73–2.18 times the per-step runtime of SOAP*. However, the additional memory requirement is negligible in the measured settings. For example, peak allocated memory on Wave1d-C is 148.6 MiB for CHIME and 148.7 MiB for SOAP*, while on HNd it is 1364.8 and 1368.5 MiB, respectively. Appendix D.3 additionally reports allocated, reserved, and process-level GPU memory measurements.

W4 & Q2. Baseline coverage and experimental transparency

We have clarified the learning-rate selection protocol in the caption of Table 2. For every method, we report the better-performing learning rate among 10−310^{-3} and 10−410^{-4}. Appendix E.1 provides the results at both learning rates for each method, making the selection procedure explicit and allowing direct inspection of the individual results.

We have also added comparisons with ENGD and SNGD in Appendix C.12. These methods are evaluated on all four forward problems under wall-clock budgets calibrated to 50,000 CHIME steps. CHIME achieves substantially lower errors on Wave1d-C and Poisson2d-C, while ENGD achieves competitive or better results on HNd and Volterra1d. Appendix D.3 further reports their computational time and memory requirements.

Finally, the Poisson2d-C plateau is now investigated through the hole-flux diagnostics and controlled metric interventions described above, providing a physical explanation for the observed performance gap.


We sincerely thank you again for the time and effort invested in this review. Your comments prompted us to go beyond aggregate benchmark comparisons and examine the individual algorithmic contributions, the precise scope of our theoretical guarantees, and the physical and computational mechanisms underlying the experimental results. We believe these revisions have substantially strengthened both the technical clarity and empirical support of the paper.

Dear Reviewer 5PSA,

Thank you again for your exceptionally thorough and constructive review. Your concerns about the individual sources of CHIME's improvements, the scope of our theoretical guarantees, and the interpretation of our empirical results have been particularly valuable in guiding our revisions.

In response, we have conducted extensive additional experiments and analyses, including training-level ablations, revised theoretical analyses, and further empirical diagnostics. We have also updated the manuscript to incorporate these results and corresponding revisions, all highlighted in blue for ease of reference.

With only a few hours remaining before the November 18 (AoE) discussion deadline, we would sincerely appreciate it if you could revisit our responses and the revised manuscript, and reconsider your evaluation in light of these new results. We would be happy to address any remaining questions or concerns before the deadline.

Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.