- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a novel multi-objective optimization method for training PINNs, called DualChebyshev. The method selects the update direction based on the Chebyshev center of the dual cone formed by the objective gradients. The proposed approach is evaluated on both forward and inverse problems and compared against several existing multi-objective optimization baselines. The authors also provide a convergence analysis showing that the proposed algorithm converges to Pareto-stationary points.
Critical strengths
-
The paper is well written and easy to follow.
-
The manuscript clearly summarizes the related literature and distinguishes the proposed method from existing approaches.
-
The proposed method demonstrates strong empirical performance across the evaluated tasks.
Critical weaknesses
-
The main contribution of this work appears to be relatively incremental. While the paper proposes a new optimization algorithm for PINN training, gradient conflict has already been widely studied, and several multi-objective optimization methods have been introduced to address this issue. Moreover, the paper does not provide an analysis of any critical issues or new phenomenon that motivates the proposed approach. Instead, it mainly introduces another variant of gradient-based multi-objective optimization and reports relatively modest improvements over existing baselines. I appreciate the authors' effort in developing and analyzing the method, but the current results do not yet demonstrate a sufficiently substantial conceptual or practical advance over existing approaches.
-
Multi-objective optimization methods generally require multiple backward passes compared with standard Adam training, resulting in higher computational cost. Although the authors report wall-clock time, an additional analysis of GPU memory usage and comparisons under a matched computational budget would help better establish the practical advantage of the proposed method.
Questions for authors
-
The choice of learning rates, ({0.001, 0.0003}), appears somewhat arbitrary. Is there a particular reason for choosing this range rather than, for example, ({0.001, 0.0001})?
-
Can the authors provide comparisons under a matched computational budget, considering factors such as wall-clock time and GPU memory usage?
5 comments
W1 (continued)
-
Practical value of the general-norm formulation. The formulation also provides useful direction-selection choices beyond . We conducted additional inverse PINN experiments under three observation settings, using three seeds and 30,000 training steps per run. The table below reports coefficient-reconstruction relative errors as mean (standard deviation).
Norm parameter Standard Higher noise Fewer observations 0.0446 (0.0047) 0.0557 (0.0053) 0.0464 (0.0049) 0.0604 (0.0004) 0.1208 (0.0738) 0.0695 (0.0024) Among all tested norms, achieves the lowest mean coefficient error in all three settings, with reductions of 26.3%, 53.9%, and 33.3% relative to , respectively. We have added the full norm sweep and experimental settings to Appendix E.2, alongside the existing forward experiments. Appendix E.3 provides another application: choosing matches the perturbation-update constraint and improves attack success rates over the evaluated Euclidean alternatives. These results demonstrate practical uses of the generalization, without suggesting that one norm is uniformly best.
-
Practical performance under a matched time budget. We also evaluated our default method under the same wall-clock budget as the competing methods. It achieves lower mean errors than standard Adam on all five forward PDE benchmarks and the lowest mean errors among the evaluated methods on three. These comparisons account for the additional cost per training step. We provide the accuracy results and GPU memory measurements in our response to W2 & Q2.
Q1: Choice of learning rates.
-
Why we chose these candidates. Our choice follows the common practice of exploring learning rates on a logarithmic scale. Approximately threefold spacing provides an intermediate value between successive powers of ten, as in . In our initial training runs, the best results among the learning rates we tried were obtained with either or . We therefore retained these two candidates for the main experiments, selecting the learning rate separately for each method and benchmark.
-
Additional validation with an expanded search range. To address your concern, we repeated the main forward comparison using three seeds and the expanded candidate set . Based on the mean relative error over three seeds, adding changed the selected learning rate only for LRA on Helmholtz and Heat 2D VC. For every other method–benchmark combination, the previously selected learning rate of either or remained unchanged.
In these two cases, selecting reduced the mean errors from 0.999978 to 0.999951 and from 0.999928 to 0.999905, respectively. Both reductions were below 0.003%, and the errors remained close to 1, so the smaller learning rate did not resolve the poor solution accuracy in either setting. The expanded search therefore provides additional evidence that excluding did not materially disadvantage the compared methods in these experiments.
W2 & Q2: Accuracy and GPU memory usage under a matched computational budget.
We agree that comparisons should account for the additional cost of computing gradients for individual objectives. We therefore conducted experiments with a matched wall-clock budget and measured peak GPU memory usage. These results and the experimental settings have been added to Appendix E.5.
-
Comparison under a matched time budget. We evaluated all five forward PDE benchmarks using three random seeds per method. For each benchmark, we first measured the mean time required for Adam to complete 50,000 training steps across three seeds, then fixed this as the common time budget for all methods. Adam was also rerun under this budget. There was no separate limit on the number of training steps.
We used the same network architecture and numbers of training points as in the original forward experiments. Each method retained its previously selected learning rate for each benchmark. Our method used with the exact solver. For standard Adam, parameter gradients were computed directly from the summed loss, without separately computing each objective's parameter gradient. The time budget included training, intermediate evaluation, and logging, with evaluation intervals matched by elapsed time.
-
Accuracy within the same time budget. The table below reports the final relative error as mean (standard deviation) over three seeds. We evaluated the model at the end of the time budget rather than selecting the best intermediate checkpoint. The common budgets were 301.5, 570.2, 269.4, 334.4, and 407.7 seconds for Helmholtz, Kovasznay, Burgers, Heat 2D VC, and Klein–Gordon, respectively. Bold values indicate the lowest mean error for each benchmark.
Method Helmholtz Kovasznay Burgers Heat 2D VC Klein–Gordon Adam 0.0284 (0.0136) 0.0051 (0.0020) 0.1686 (0.0363) 3.3535 (0.0225) 0.0288 (0.0052) LRA 1.0000 (0.0000) 0.0074 (0.0020) 0.4063 (0.0214) 1.0000 (0.0002) 1.0379 (0.0153) ReLoBRaLo 0.0713 (0.0275) 0.0058 (0.0010) 0.1935 (0.0054) 2.7673 (0.3570) 0.0437 (0.0198) MGDA 0.6324 (0.2530) 0.0119 (0.0024) 0.2655 (0.0607) 0.9256 (0.0229) 0.2603 (0.0275) PCGrad 0.0234 (0.0222) 0.0046 (0.0010) 0.1258 (0.0553) 0.4086 (0.0687) 0.0719 (0.0556) IMTL-G 0.0138 (0.0089) 0.0046 (0.0013) 0.2473 (0.0264) 1.3119 (0.2900) 0.0767 (0.0627) ConFIG — — 0.1402 (0.0388) 0.2852 (0.0943) 0.0543 (0.0361) HARMONIC 0.0117 (0.0026) 0.0066 (0.0044) 0.1363 (0.0255) 0.2619 (0.0184) 0.0271 (0.0056) GAPO 0.0064 (0.0007) 0.0046 (0.0019) 0.1096 (0.0385) 0.2324 (0.0360) 0.0320 (0.0253) GAPO 0.0231 (0.0067) 0.0938 (0.0276) 0.1191 (0.0070) 0.7363 (0.1070) 0.0722 (0.0321) Ours 0.0133 (0.0064) 0.0025 (0.0004) 0.0942 (0.0148) 0.2429 (0.0376) 0.0168 (0.0089) A dash denotes an omitted entry: ConFIG is omitted for the two-objective benchmarks, where it coincides with our method.
Our method achieves lower mean errors than Adam on all five benchmarks, despite completing fewer training steps within the same time budget. It also achieves the lowest mean errors among the evaluated methods on Kovasznay, Burgers, and Klein–Gordon. GAPO with achieves the lowest mean errors on Helmholtz and Heat 2D VC. Thus, the additional cost per step does not eliminate the practical accuracy benefits of our method, although it does not perform best on every benchmark.
-
GPU memory usage. We measured peak GPU allocated memory during training, including intermediate evaluation, with the same network architecture and numbers of training points across methods. The table below reports the mean of the per-run peaks over three seeds, in MiB.
Method Helmholtz Kovasznay Burgers Heat 2D VC Klein–Gordon Adam 48.706 78.745 43.074 38.418 43.176 ReLoBRaLo 80.335 144.159 51.811 52.449 56.994 Other methods* 80.333 144.157 51.808 52.446 56.991 Ours 80.333 144.157 51.808 52.446 56.991 Other methods: LRA, MGDA, PCGrad, IMTL-G, HARMONIC, GAPO with , and ConFIG on the three-objective benchmarks. These methods have matching peak values at the reported precision.
Our method shows essentially the same peak GPU allocated memory as the other methods that compute objective gradients separately in our implementation. Although this is approximately 20–83% higher than standard Adam across the five benchmarks, our method does not introduce an additional peak GPU memory burden relative to those multi-objective baselines in these experiments.
W1: Conceptual and practical contributions.
Thank you for your review. The paper's central point is that resolving gradient conflict does not by itself determine a good update direction. Requiring equal normalized progress, as ConFIG does, can drive first-order progress on every objective toward zero even when the gradients have equal magnitudes and no pairwise conflicts. Our criterion avoids this loss (Theorem 4.1), derives from a single geometric objective the design choices that GAPO, HARMONIC, and DCGD Center adopt for separate reasons, giving GAPO's direction a max–min optimality characterization absent from the original work, and yields non-Euclidean variants that reduce the mean coefficient error on inverse problems by 26–54% relative to p=2. We elaborate on each point below.
A limitation beyond gradient conflict and magnitude imbalance. To illustrate the limitation of equal normalized progress, consider three unit gradients:
g^1=(c,1/2,0),g^2=(c,−1/2,0),g^3=(b,0,δ),
c=23,b=1−δ2,0<δ<21.
These gradients are linearly independent, and all pairwise cosine similarities are at least 1/2. Nevertheless, imposing the equal-normalized-progress condition used by ConFIG produces a unit direction whose progress for every objective approaches zero as δ↓0. Our direction v⋆=(1,0,0) maintains progress of at least 3/2 for every objective. Using the notation of Theorem 4.1, the optimal values of our max–min problem and its equality-constrained counterpart are
r⋆=c,r~=δ2+(b−c)2cδ⟶0(δ↓0).
Thus, enforcing equality can make normalized first-order progress arbitrarily small for every objective, even without gradient conflict or magnitude imbalance. This identifies a concrete limitation of equal-progress direction selection and explains why relaxing equality to inequalities can matter substantially. The full construction and proof have been added to Appendix B.4.
Explaining existing algorithmic choices through a common criterion. Our analysis also clarifies why particular choices of normalization and aggregation fit together. For example, GAPO combines gradient normalization with MGDA's minimum-norm convex aggregation. GAPO motivates normalization by gradient-magnitude imbalance and shows that the resulting direction is a common descent direction, but it does not identify an optimization criterion from which normalization and minimum-norm aggregation jointly follow. To our knowledge, our analysis provides the first such criterion for this direction. Our formulation derives both choices from a single geometric objective: maximizing the minimum distance from a unit direction to the defining hyperplanes of the dual cone. Normalization follows from the distance formula, and minimum-norm convex aggregation follows from the dual problem. Thus, our formulation with p=2 recovers GAPO's direction at ρ=1, identifying what normalization and aggregation jointly optimize.
The other connections yield complementary insights. For HARMONIC, membership in the gradient-cone/dual-cone intersection is explicitly enforced; our analysis shows that this property follows from optimality when p=2. For DCGD Center, the angle-bisector construction coincides with our center criterion in the nondegenerate two-objective Euclidean case. These connections give a common geometric basis for understanding methods developed from different design motivations, making their direction-selection principles directly comparable.
We continue our response to W1 in the next comment.