acceptodds
Under review as a conference paper at ICLR 2027

Chebyshev Center-Based Direction Selection for Multi-Objective Optimization and Training PINNs

Abstract

Physics-informed neural networks (PINNs) are a promising approach for solving partial differential equations (PDEs). Their training, however, is often difficult because multiple loss terms induced by PDE residuals and boundary or initial conditions must be optimized simultaneously. To address this difficulty, existing approaches often construct update directions by explicitly enforcing particular desirable properties, such as scale robustness and simultaneous descent. While effective in many cases, such property-by-property designs can make it unclear which conditions are essential, what geometric principle determines the selected update direction, and how different methods are structurally related. In this work, we formulate update-direction selection for PINN training as a Chebyshev-center problem in the dual cone. The proposed formulation selects a normalized direction that maximizes the minimum distance to the cone facets. The resulting formulation admits an efficient dual problem in a much lower-dimensional space and yields a convergence guarantee in the nonconvex setting. It also recovers the key desirable properties targeted by existing approaches without imposing them separately; rather, they follow from the single geometric criterion underlying the formulation. This makes the selected direction interpretable through a single geometric rule and provides a unified basis for systematically comparing related direction-selection methods. Experiments on several PINN benchmarks further demonstrate strong empirical performance of the proposed method.

Primary area:
optimization
Keywords:
Multi-Objective Optimization, Chebyshev Center, PINNs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FFohOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a novel multi-objective optimization method for training PINNs, called DualChebyshev. The method selects the update direction based on the Chebyshev center of the dual cone formed by the objective gradients. The proposed approach is evaluated on both forward and inverse problems and compared against several existing multi-objective optimization baselines. The authors also provide a convergence analysis showing that the proposed algorithm converges to Pareto-stationary points.

Critical strengths
  1. The paper is well written and easy to follow.

  2. The manuscript clearly summarizes the related literature and distinguishes the proposed method from existing approaches.

  3. The proposed method demonstrates strong empirical performance across the evaluated tasks.

Critical weaknesses
  1. The main contribution of this work appears to be relatively incremental. While the paper proposes a new optimization algorithm for PINN training, gradient conflict has already been widely studied, and several multi-objective optimization methods have been introduced to address this issue. Moreover, the paper does not provide an analysis of any critical issues or new phenomenon that motivates the proposed approach. Instead, it mainly introduces another variant of gradient-based multi-objective optimization and reports relatively modest improvements over existing baselines. I appreciate the authors' effort in developing and analyzing the method, but the current results do not yet demonstrate a sufficiently substantial conceptual or practical advance over existing approaches.

  2. Multi-objective optimization methods generally require multiple backward passes compared with standard Adam training, resulting in higher computational cost. Although the authors report wall-clock time, an additional analysis of GPU memory usage and comparisons under a matched computational budget would help better establish the practical advantage of the proposed method.

Questions for authors
  1. The choice of learning rates, ({0.001, 0.0003}), appears somewhat arbitrary. Is there a particular reason for choosing this range rather than, for example, ({0.001, 0.0001})?

  2. Can the authors provide comparisons under a matched computational budget, considering factors such as wall-clock time and GPU memory usage?

5 comments

W1: Conceptual and practical contributions.

Thank you for your review. The paper's central point is that resolving gradient conflict does not by itself determine a good update direction. Requiring equal normalized progress, as ConFIG does, can drive first-order progress on every objective toward zero even when the gradients have equal magnitudes and no pairwise conflicts. Our criterion avoids this loss (Theorem 4.1), derives from a single geometric objective the design choices that GAPO, HARMONIC, and DCGD Center adopt for separate reasons, giving GAPO's direction a max–min optimality characterization absent from the original work, and yields non-Euclidean variants that reduce the mean coefficient error on inverse problems by 26–54% relative to p=2p = 2. We elaborate on each point below.

  • A limitation beyond gradient conflict and magnitude imbalance. To illustrate the limitation of equal normalized progress, consider three unit gradients:

    g^1=(c,1/2,0),g^2=(c,−1/2,0),g^3=(b,0,δ),\hat{g}_1 = (c, 1/2, 0), \qquad \hat{g}_2 = (c, -1/2, 0), \qquad \hat{g}_3 = (b, 0, \delta),

    c=32,b=1−δ2,0<δ<12.c = \frac{\sqrt{3}}{2}, \qquad b = \sqrt{1 - \delta^2}, \qquad 0 < \delta < \frac{1}{2}.

    These gradients are linearly independent, and all pairwise cosine similarities are at least 1/21/2. Nevertheless, imposing the equal-normalized-progress condition used by ConFIG produces a unit direction whose progress for every objective approaches zero as δ↓0\delta \downarrow 0. Our direction v⋆=(1,0,0)v^\star = (1, 0, 0) maintains progress of at least 3/2\sqrt{3}/2 for every objective. Using the notation of Theorem 4.1, the optimal values of our max–min problem and its equality-constrained counterpart are

    r⋆=c,r~=cδδ2+(b−c)2⟶0(δ↓0).r^\star = c, \qquad \tilde{r} = \frac{c\delta}{\sqrt{\delta^2 + (b - c)^2}} \longrightarrow 0 \quad (\delta \downarrow 0).

    Thus, enforcing equality can make normalized first-order progress arbitrarily small for every objective, even without gradient conflict or magnitude imbalance. This identifies a concrete limitation of equal-progress direction selection and explains why relaxing equality to inequalities can matter substantially. The full construction and proof have been added to Appendix B.4.

  • Explaining existing algorithmic choices through a common criterion. Our analysis also clarifies why particular choices of normalization and aggregation fit together. For example, GAPO combines gradient normalization with MGDA's minimum-norm convex aggregation. GAPO motivates normalization by gradient-magnitude imbalance and shows that the resulting direction is a common descent direction, but it does not identify an optimization criterion from which normalization and minimum-norm aggregation jointly follow. To our knowledge, our analysis provides the first such criterion for this direction. Our formulation derives both choices from a single geometric objective: maximizing the minimum distance from a unit direction to the defining hyperplanes of the dual cone. Normalization follows from the distance formula, and minimum-norm convex aggregation follows from the dual problem. Thus, our formulation with p=2p = 2 recovers GAPO's direction at ρ=1\rho = 1, identifying what normalization and aggregation jointly optimize.

    The other connections yield complementary insights. For HARMONIC, membership in the gradient-cone/dual-cone intersection is explicitly enforced; our analysis shows that this property follows from optimality when p=2p = 2. For DCGD Center, the angle-bisector construction coincides with our center criterion in the nondegenerate two-objective Euclidean case. These connections give a common geometric basis for understanding methods developed from different design motivations, making their direction-selection principles directly comparable.

We continue our response to W1 in the next comment.

W1 (continued)

  • Practical value of the general-norm formulation. The formulation also provides useful direction-selection choices beyond p=2p = 2. We conducted additional inverse PINN experiments under three observation settings, using three seeds and 30,000 training steps per run. The table below reports coefficient-reconstruction relative errors as mean (standard deviation).

    Norm parameterStandardHigher noiseFewer observations
    p=1p = 10.0446 (0.0047)0.0557 (0.0053)0.0464 (0.0049)
    p=2p = 20.0604 (0.0004)0.1208 (0.0738)0.0695 (0.0024)

    Among all tested norms, p=1p = 1 achieves the lowest mean coefficient error in all three settings, with reductions of 26.3%, 53.9%, and 33.3% relative to p=2p = 2, respectively. We have added the full norm sweep and experimental settings to Appendix E.2, alongside the existing forward experiments. Appendix E.3 provides another application: choosing p=1p = 1 matches the ℓ∞\ell_\infty perturbation-update constraint and improves attack success rates over the evaluated Euclidean alternatives. These results demonstrate practical uses of the generalization, without suggesting that one norm is uniformly best.

  • Practical performance under a matched time budget. We also evaluated our default p=2p = 2 method under the same wall-clock budget as the competing methods. It achieves lower mean errors than standard Adam on all five forward PDE benchmarks and the lowest mean errors among the evaluated methods on three. These comparisons account for the additional cost per training step. We provide the accuracy results and GPU memory measurements in our response to W2 & Q2.

Q1: Choice of learning rates.

  • Why we chose these candidates. Our choice follows the common practice of exploring learning rates on a logarithmic scale. Approximately threefold spacing provides an intermediate value between successive powers of ten, as in 10−3, 3×10−4, 10−410^{-3},\ 3 \times 10^{-4},\ 10^{-4}. In our initial training runs, the best results among the learning rates we tried were obtained with either 10−310^{-3} or 3×10−43 \times 10^{-4}. We therefore retained these two candidates for the main experiments, selecting the learning rate separately for each method and benchmark.

  • Additional validation with an expanded search range. To address your concern, we repeated the main forward comparison using three seeds and the expanded candidate set {10−3,3×10−4,10−4}\lbrace 10^{-3}, 3 \times 10^{-4}, 10^{-4}\rbrace. Based on the mean relative L2L^2 error over three seeds, adding 10−410^{-4} changed the selected learning rate only for LRA on Helmholtz and Heat 2D VC. For every other method–benchmark combination, the previously selected learning rate of either 10−310^{-3} or 3×10−43 \times 10^{-4} remained unchanged.

    In these two cases, selecting 10−410^{-4} reduced the mean errors from 0.999978 to 0.999951 and from 0.999928 to 0.999905, respectively. Both reductions were below 0.003%, and the errors remained close to 1, so the smaller learning rate did not resolve the poor solution accuracy in either setting. The expanded search therefore provides additional evidence that excluding 10−410^{-4} did not materially disadvantage the compared methods in these experiments.

W2 & Q2: Accuracy and GPU memory usage under a matched computational budget.

We agree that comparisons should account for the additional cost of computing gradients for individual objectives. We therefore conducted experiments with a matched wall-clock budget and measured peak GPU memory usage. These results and the experimental settings have been added to Appendix E.5.

  • Comparison under a matched time budget. We evaluated all five forward PDE benchmarks using three random seeds per method. For each benchmark, we first measured the mean time required for Adam to complete 50,000 training steps across three seeds, then fixed this as the common time budget for all methods. Adam was also rerun under this budget. There was no separate limit on the number of training steps.

    We used the same network architecture and numbers of training points as in the original forward experiments. Each method retained its previously selected learning rate for each benchmark. Our method used p=2p = 2 with the exact solver. For standard Adam, parameter gradients were computed directly from the summed loss, without separately computing each objective's parameter gradient. The time budget included training, intermediate evaluation, and logging, with evaluation intervals matched by elapsed time.

  • Accuracy within the same time budget. The table below reports the final relative L2L^2 error as mean (standard deviation) over three seeds. We evaluated the model at the end of the time budget rather than selecting the best intermediate checkpoint. The common budgets were 301.5, 570.2, 269.4, 334.4, and 407.7 seconds for Helmholtz, Kovasznay, Burgers, Heat 2D VC, and Klein–Gordon, respectively. Bold values indicate the lowest mean error for each benchmark.

    MethodHelmholtzKovasznayBurgersHeat 2D VCKlein–Gordon
    Adam0.0284 (0.0136)0.0051 (0.0020)0.1686 (0.0363)3.3535 (0.0225)0.0288 (0.0052)
    LRA1.0000 (0.0000)0.0074 (0.0020)0.4063 (0.0214)1.0000 (0.0002)1.0379 (0.0153)
    ReLoBRaLo0.0713 (0.0275)0.0058 (0.0010)0.1935 (0.0054)2.7673 (0.3570)0.0437 (0.0198)
    MGDA0.6324 (0.2530)0.0119 (0.0024)0.2655 (0.0607)0.9256 (0.0229)0.2603 (0.0275)
    PCGrad0.0234 (0.0222)0.0046 (0.0010)0.1258 (0.0553)0.4086 (0.0687)0.0719 (0.0556)
    IMTL-G0.0138 (0.0089)0.0046 (0.0013)0.2473 (0.0264)1.3119 (0.2900)0.0767 (0.0627)
    ConFIG——0.1402 (0.0388)0.2852 (0.0943)0.0543 (0.0361)
    HARMONIC0.0117 (0.0026)0.0066 (0.0044)0.1363 (0.0255)0.2619 (0.0184)0.0271 (0.0056)
    GAPO ρ=1\rho = 10.0064 (0.0007)0.0046 (0.0019)0.1096 (0.0385)0.2324 (0.0360)0.0320 (0.0253)
    GAPO ρ=2\rho = 20.0231 (0.0067)0.0938 (0.0276)0.1191 (0.0070)0.7363 (0.1070)0.0722 (0.0321)
    Ours0.0133 (0.0064)0.0025 (0.0004)0.0942 (0.0148)0.2429 (0.0376)0.0168 (0.0089)

    A dash denotes an omitted entry: ConFIG is omitted for the two-objective benchmarks, where it coincides with our method.

    Our method achieves lower mean errors than Adam on all five benchmarks, despite completing fewer training steps within the same time budget. It also achieves the lowest mean errors among the evaluated methods on Kovasznay, Burgers, and Klein–Gordon. GAPO with ρ=1\rho = 1 achieves the lowest mean errors on Helmholtz and Heat 2D VC. Thus, the additional cost per step does not eliminate the practical accuracy benefits of our method, although it does not perform best on every benchmark.

  • GPU memory usage. We measured peak GPU allocated memory during training, including intermediate evaluation, with the same network architecture and numbers of training points across methods. The table below reports the mean of the per-run peaks over three seeds, in MiB.

    MethodHelmholtzKovasznayBurgersHeat 2D VCKlein–Gordon
    Adam48.70678.74543.07438.41843.176
    ReLoBRaLo80.335144.15951.81152.44956.994
    Other methods*80.333144.15751.80852.44656.991
    Ours80.333144.15751.80852.44656.991

    Other methods: LRA, MGDA, PCGrad, IMTL-G, HARMONIC, GAPO with ρ=1,2\rho = 1, 2, and ConFIG on the three-objective benchmarks. These methods have matching peak values at the reported precision.

    Our method shows essentially the same peak GPU allocated memory as the other methods that compute objective gradients separately in our implementation. Although this is approximately 20–83% higher than standard Adam across the five benchmarks, our method does not introduce an additional peak GPU memory burden relative to those multi-objective baselines in these experiments.

Reviewer_rZPVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DualChebyshev, a multi-objective optimization direction selection method based on the Chebyshev center, mainly applied to the training of Physics-Informed Neural Networks (PINNs). The authors determine the optimization direction by maximizing the minimum distance from the normalized update direction to the boundary of the dual cone, and derive the corresponding low-dimensional dual problem, primal direction recovery formula, and non-convex convergence guarantees. In addition, the paper establishes the connections between the proposed method and DCGD, HARMONIC, IMTL-G, ConFIG, MGDA, and GAPO through geometric analysis, and conducts experimental validation on forward and inverse PDE problems.

Critical strengths
  1. The paper interprets direction selection as a max-min distance problem in the dual cone, providing a unified perspective for understanding gradient normalization, common descent direction, and the balance between objectives. This interpretation has a certain theoretical value.
  2. The mathematical framework of this paper is relatively complete, not only giving the primal problem and its low-dimensional dual form, but also discussing the recovery of the optimal direction, Pareto stationary points, and non-convex optimization convergence. The appendix contains relatively detailed derivations, enhancing the verifiability of the theoretical part.
  3. The authors not only tested five forward PDE problems and three inverse problem settings, but also provided GAPO comparison, norm selection ablation, adversarial attack, without-Adam update, and runtime analysis in the appendix. These supplementary experiments forcefully enhance the reliability of the argument.
Critical weaknesses
  1. Just as the authors argued in Section 4, under the Euclidean norm setting, the direction selection of this paper is consistent with DCGD in the two-objective case, and its dual problem is also equivalent to GAPO (ρ=1\rho=1). Therefore, the novelty of the optimization method proposed in this paper is relatively incremental. However, considering that the contribution of the paper is mainly theoretical analysis, I think this is acceptable.
Questions for authors

In the two-objective case, the proposed method appears to be mathematically equivalent to DCGD (Center), including the final update scaling. However, Table 1 reports different performance between the two methods (on Helmholtz and Kovasznay). Therefore, the performance advantages of the proposed method do not seem to be explained by methodological differences. Could the authors clarify which DCGD variant was implemented and explain the observed differences, particularly under matched learning rates and random seeds?

3 comments

Thank you for your careful review and for recognizing the value of our theoretical analysis. We address your concern about novelty and your question about the DCGD comparison below.

W1: Algorithmic novelty and analytical contributions.

  • What the connections to existing methods explain. The fact that our method recovers existing direction-selection rules when p=2p = 2 also helps explain the principles underlying those rules. For example, reducing gradient imbalance motivates GAPO's use of normalization, but this motivation alone does not explain why the normalized gradients should be combined through MGDA's minimum-norm convex aggregation. Our analysis shows that both normalization and this aggregation rule follow from a single geometric criterion: maximizing the minimum distance from a unit direction to the defining hyperplanes of the dual cone. Thus, recovering GAPO's direction explains what the combination jointly optimizes and provides a geometric justification for pairing normalization with this particular aggregation rule.

    Similarly, DCGD Center's angle-bisector construction coincides with the Chebyshev-center direction in the nondegenerate two-objective Euclidean case. This gives the Center construction an explicit geometric optimality interpretation, consistent with its reported empirical advantage over other DCGD variants.

  • Practical relevance beyond the Euclidean case. The general-pp formulation also provides practically useful choices. Appendix E.3 demonstrates improved attack success rates when p=1p = 1 matches the ℓ∞\ell_\infty perturbation-update constraint.

    Within PINNs, our additional inverse experiments show that p=1p = 1 achieves the lowest mean coefficient-reconstruction error among the tested norms in all three observation settings. Averaged over three seeds, the errors are 26.3%, 53.9%, and 33.3% lower than those at p=2p = 2 under the standard, higher-noise, and fewer-observation settings, respectively. We have added these inverse results and their experimental settings to Appendix E.2. They complement the existing forward experiments in that appendix and demonstrate the practical relevance of the generalization alongside its analytical contribution.

Q1: Equivalence to DCGD Center and the reported performance differences.

  • Implemented variant and mathematical equivalence. You are correct that, for p=2p = 2 in the nondegenerate two-objective case, our method is mathematically equivalent to DCGD Center, including the final scaling. The variant used in our experiments was DCGD Center, reimplemented within the same training and Adam framework as the other methods.

  • Numerical implementation differences. The two methods were implemented through separate numerical computation paths. For example, our implementation summed the inner products of the individual gradients with the unit direction, whereas the DCGD Center implementation first summed the gradients and then computed the projection. These algebraically equivalent computations can differ in floating-point arithmetic. Diagnostic comparisons with matched initialization, training points, and learning rates confirmed that small numerical differences between the implementations can lead to diverging training trajectories.

    In addition, the reported Kovasznay results used different selected learning rates: 3×10−43 \times 10^{-4} for DCGD and 10−310^{-3} for our method, as listed in Table 3 of Appendix D.2 in the original submission. These observations do not establish the full cause of the original performance gaps. Regardless, those gaps should not be interpreted as an algorithmic advantage over DCGD Center.

  • Corrections to the experimental comparison. We have removed the DCGD row from the forward benchmark results in Table 1 (Section 5.1), along with the corresponding entries in the learning-rate table (Table 3, Appendix D.2) and the forward runtime table (Table 10 in the original submission, now Table 11 in Appendix E.5). DCGD was evaluated only on the two-objective problems where the methods are mathematically equivalent. We retain the theoretical comparison in Section 4 and have clarified that the equivalence includes the final scaling. We appreciate your pointing out this inconsistency in the experimental comparison.

I thank the authors for their clarifications. I am maintaining my positive rating.

We thank you again for your helpful feedback and positive assessment of our work.

Reviewer_4ht1OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Chebyshev center-based framework for selecting update directions in multi-objective optimization, with a particular focus on training physics-informed neural networks (PINNs).

Critical strengths
  1. Competitive empirical performance.
  2. A well-structured primal-dual formulation.
Critical weaknesses
  1. Limited algorithmic novelty relative to existing normalized-gradient methods. Concern is that the practical algorithmic contribution appears considerably smaller than the geometric presentation suggests.
Questions for authors
  1. Since the proposed dual problem with (p=2) is equivalent to GAPO with (\rho=1), what is the precise algorithmic contribution beyond the geometric interpretation and adaptive scalar?

  2. How much of the empirical performance gain is attributable specifically to the adaptive scalar? Does it remain beneficial when the resulting direction is passed through Adam?

  3. Can the authors provide more extensive experiments with non-Euclidean norms on PINN benchmarks and clarify the computational trade-offs of the endpoint cases?

4 comments

Thank you for your careful review and constructive questions. Below, we clarify the algorithmic and analytical contributions and address your questions about adaptive scaling, non-Euclidean norms, and their computational costs.

W1 & Q1: Algorithmic novelty and analytical contributions.

At p=2p = 2, our method selects the same direction as GAPO with ρ=1\rho = 1. However, beyond the geometric interpretation and the adaptive scalar, the algorithmic contribution is the family of direction-selection rules for p≠2p \neq 2. GAPO cannot produce these directions: its exponent ρ\rho rescales the gradients inside a Euclidean minimum-norm problem, whereas pp jointly changes the normalization, the norm of the dual problem, and the map that recovers the update direction. With p=1p = 1, this lowers the mean coefficient error on the inverse PINN problem by 26–54% relative to p=2p = 2 and raises the simultaneous attack success rate under an ℓ∞\ell_\infty budget from 9.47% to 13.07%. The formulation also yields a nonconvex convergence guarantee for the scaled update (Theorem 3.8) and per-objective dominance over ConFIG's equality-constrained direction (Theorem 4.1). We detail these points below.

  • Practical relevance of the general-pp formulation. The general-pp formulation allows direction selection to reflect the geometry of the constraints on an update. For example, in the adversarial-attack experiment in Appendix E.3, choosing p=1p = 1 yields a primal direction constraint in the ℓ∞\ell_\infty norm, matching the perturbation-update constraint. This formulation achieves higher attack success rates than the evaluated Euclidean alternatives, illustrating a concrete benefit of choosing a norm suited to the application.

    Within PINNs, Appendix E.2 reports lower mean errors for non-Euclidean choices on some forward benchmarks. Our additional inverse experiments further show that p=1p = 1 achieves the lowest mean coefficient-reconstruction error among the tested norms in all three observation settings. We have added these inverse results and their experimental settings to Appendix E.2 and summarize the findings in our response to Q3.

  • A geometric justification for normalization and aggregation. GAPO combines MGDA with gradient normalization to address gradient imbalance. Reducing the influence of gradient magnitudes motivates normalization, but does not by itself explain why the normalized gradients should be combined through MGDA's minimum-norm convex aggregation.

    Our formulation connects these two choices through a single geometric objective: selecting a unit direction that maximizes the minimum distance to the defining hyperplanes of the dual cone. Normalization follows from the formula for these distances, and minimum-norm convex aggregation follows from the dual problem. For p=2p = 2, this recovers GAPO's direction at ρ=1\rho = 1. Thus, the equivalence explains what the two operations jointly optimize and provides a geometric justification for their combination.

  • The cost of enforcing equal normalized progress. The primal formulation allows us to examine the additional restrictions imposed by other constructions. For ConFIG, we relate its equal-normalized-progress requirement to equality constraints in our p=2p = 2 max–min formulation. When ConFIG's linear system is consistent, our unit direction v⋆v^\star achieves at least as much normalized first-order progress as ConFIG's unit direction v~\tilde{v} for every objective:

    g^iTv⋆≥g^iTv~for every i,g^i=gi∥gi∥2(Theorem 4.1).\hat{g}_i^{\mathsf{T}} v^\star \ge \hat{g}_i^{\mathsf{T}} \tilde{v} \quad \text{for every } i, \qquad \hat{g}_i = \frac{g_i}{\lVert g_i\rVert_2} \quad (\text{Theorem 4.1}).

    Our additional example shows that this restriction can have a substantial cost even for linearly independent unit gradients with no pairwise conflicts. Using the notation of Theorem 4.1, let (v⋆,r⋆)(v^\star, r^\star) and (v~,r~)(\tilde{v}, \tilde{r}) denote the optimal solutions of (P) and its equality-constrained counterpart (P=)(\mathrm{P}_=), respectively. The example gives

    r⋆=c,r~=cδδ2+(b−c)2⟶0(δ↓0),r^\star = c, \qquad \tilde{r} = \frac{c\delta}{\sqrt{\delta^2 + (b - c)^2}} \longrightarrow 0 \quad (\delta \downarrow 0),

    c=32,b=1−δ2,0<δ<12.c = \frac{\sqrt{3}}{2}, \quad b = \sqrt{1 - \delta^2}, \quad 0 < \delta < \frac{1}{2}.

    All objectives have progress r~\tilde{r} under the equality constraint, whereas our direction maintains progress of at least 3/2\sqrt{3}/2 for every objective. Thus, relaxing equality can prevent a severe reduction in normalized first-order progress even when neither gradient conflict nor magnitude imbalance is present. The full construction and proof have been added to Appendix B.4.

We continue our response to W1 & Q1 in the next comment.

W1 & Q1 (continued)

  • Gradient-cone membership follows from optimality. Our geometric analysis also connects HARMONIC's use of the intersection of the gradient cone K\mathbf{K} and its dual cone K∗\mathbf{K}^* to the Chebyshev-center criterion. When p=2p = 2, our optimal direction satisfies

    v⋆∈K∩K∗v^\star \in \mathbf{K} \cap \mathbf{K}^*

    without imposing membership in the gradient cone as an additional constraint. This identifies a property that follows naturally from our optimality criterion.

Together, these results establish both the practical value of direction selection beyond p=2p = 2 and the analytical value of explaining existing design choices, their restrictions, and their consequences.

Q2: Effect of adaptive scaling and its interaction with Adam.

  • Isolating the effect of scaling. The GAPO comparison in Appendix E.1 directly examines the effect of scaling when both methods select the same direction at a given parameter point. Specifically, GAPO with ρ=1\rho = 1 and our method with p=2p = 2 yield the same dual aggregate wtw_t and unit direction vt=wt/∥wt∥2v_t = w_t / \lVert w_t\rVert_2. GAPO passes wtw_t to Adam, whereas our method passes the scaled vector dtd_t. For wt≠0w_t \neq 0,

    \qquad a_t = \frac{\sum_i g_{i,t}^{\mathsf{T}} v_t}{\lVert w_t\rVert_2}.$$ Thus, $a_t$ is the relative scaling factor between the two inputs to Adam. This scaling also gives $\nabla\mathcal{L}(\theta_t)^{\mathsf T}d_t=\lVert d_t\rVert_q^2$, where $\mathcal{L}=\sum_i\mathcal{L}_i$, a relation used in the convergence proof for the direct update in Theorem 3.8.
  • Why adaptive scaling can affect Adam. Although Adam cancels a fixed positive rescaling of its entire input sequence when its numerical stabilizer ϵ\epsilon is ignored, this cancellation does not generally hold for a time-varying factor ata_t. Denoting Adam's first and second moments by MtM_t and VtV_t, our scaled input gives

    \qquad V_t = \beta_2 V_{t-1} + (1 - \beta_2) a_t^2 (w_t \odot w_t).$$ These moments accumulate past inputs with their respective time-varying scaling factors, so a common factor cannot generally be removed from the bias-corrected update. Adaptive scaling therefore changes how past gradients contribute to Adam's moments and can alter the resulting updates and optimization trajectory.
  • Empirical effect with Adam. Table 5 in Appendix E.1 compares both methods using Adam. On Kovasznay, our scaling reduces the mean relative error from 0.0055 to 0.0017, an approximately 69% reduction. The differences are smaller on the other benchmarks, and the scaling does not improve performance uniformly. These results show that adaptive scaling can remain beneficial even when the selected direction is passed through Adam. Without Adam, the direct-update experiments in Appendix E.4 also show a lower mean error on Helmholtz for our method than for GAPO with ρ=1\rho=1 (0.1613 versus 0.9988) under the tested learning rates and training budget.

Q3: Additional non-Euclidean experiments and endpoint computational costs.

  • Additional inverse experiments. We conducted additional experiments on the inverse PINN benchmark under three observation settings: standard, higher noise, and fewer observations. We evaluated p∈{1,1.5,2,3,4,∞}p \in \lbrace 1, 1.5, 2, 3, 4, \infty \rbrace using three random seeds per setting. All runs completed 30,000 training steps without early stopping. All entries, including p=2p=2, were obtained from separate runs for this norm comparison on a different server with a different GPU model from that used for the main inverse experiments. The table below reports the coefficient-reconstruction relative error as mean (standard deviation). These results and their experimental settings have also been added to Appendix E.2.

    ppStandardHigher noiseFewer observations
    10.0446 (0.0047)0.0557 (0.0053)0.0464 (0.0049)
    1.50.0466 (0.0050)0.0744 (0.0072)0.0557 (0.0036)
    20.0604 (0.0004)0.1208 (0.0738)0.0695 (0.0024)
    30.1328 (0.0956)0.1287 (0.0955)0.1752 (0.0663)
    40.1865 (0.0656)0.2169 (0.0218)0.1761 (0.0804)
    ∞\infty0.2390 (0.0000)0.2390 (0.0000)0.2390 (0.0000)

    Standard deviations displayed as 0.0000 are rounded values.

  • Improved coefficient recovery with non-Euclidean norms. Among the tested norms, p=1p = 1 achieves the lowest mean coefficient-reconstruction error in all three settings, with reductions of 26.3%, 53.9%, and 33.3% relative to p=2p = 2, respectively. Across all three settings, the mean coefficient-reconstruction error increases monotonically over the tested values of pp. These results provide further evidence that non-Euclidean choices can improve coefficient recovery in inverse problems solved with PINNs. Note that the nearly identical errors at p=∞p = \infty correspond to an approximately constant coefficient estimate a(x,y)≈2a(x, y) \approx 2, which fails to recover the spatial variation in the interior.

  • Training time for the endpoint cases. To examine the computational trade-offs of the endpoint cases, we separately measured the training time and its breakdown for p=1p = 1 and p=∞p = \infty with our CPU LP implementation and p=2p = 2 with the exact solver. The following measurements are from the standard inverse setting with one seed, using 30,000 steps for each method. These timing runs were conducted on a different server with a different GPU model from that used for the full-training runtimes reported for the main inverse experiments.

    MethodTotal training time (min)Solver time per step (ms)Solver share of total time
    p=2p = 2, exact8.030.6434.0%
    p=1p = 1, LP33.3546.10869.1%
    p=∞p = \infty, LP54.9086.15378.5%
  • Computational trade-offs. In these runs, p=1p = 1 and p=∞p = \infty require approximately 4.2 and 6.8 times the training time of p=2p = 2, respectively. LP solving accounts for 69.1–78.5% of total training time and is the main bottleneck in the current endpoint implementation. All other direction-computation operations, including CPU–GPU transfers, together account for only 2.3% and 1.9%, respectively.

    Together with the coefficient-recovery results above, these measurements show a clear accuracy–time trade-off for p=1p = 1: it achieves lower coefficient errors at the cost of longer training. By contrast, p=∞p = \infty incurs a higher cost without improving coefficient recovery, while p=2p = 2 remains the faster choice with the exact solver.

Dear Reviewer 4ht1,

Thank you for your careful review. We have addressed each of your comments and questions in our rebuttal and would be grateful if you could take a look when you have a chance.

Thanks, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.