acceptodds
Under review as a conference paper at ICLR 2027

Gradient–Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks

Abstract

Training Physics-Informed Neural Networks (PINNs) requires jointly optimizing physics residual and initial/boundary condition loss terms, which often induce conflicting gradients. Gradient surgery methods mitigate this issue by constructing directions from loss-specific gradients to reduce conflict before optimizer transformation. However, even when the constructed direction is conflict-free, this property may not be preserved after optimizer transformation. Let denote the direction constructed by gradient surgery, the optimizer proposal, and the conflict-free cone induced by the loss-specific gradients. We show that modern optimizers can transform through mechanisms including historical state, adaptive scaling, preconditioning, and decoupled weight decay, so does not generally imply . We refer to this optimizer-induced discrepancy in conflict-freeness between and as **Gradient-Update Mismatch** (GUM). Accordingly, we propose **Gradient-Update Alignment** (GUA), which performs conflict handling on the update actually applied to the parameters. Specifically, after the optimizer transforms into , GUA projects onto to obtain the aligned update , which is then applied to the parameters. For optimizers with internal state, GUA further aligns the state with targets derived from the applied update. Extensive experiments show that GUM is widespread across diverse optimizers, with conflict rates reaching up to 86.3%. Across all PINN settings, GUA achieves conflict-free applied updates and consistently improves various gradient surgery methods, reducing relative error by up to 98.2%. Data and code are available at [https://anonymous.4open.science/r/GUA-F010](https://anonymous.4open.science/r/GUA-F010).

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Physics Informed Neural Networks, Conflicting gradients

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xhuCOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on the Gradient Update Mismatch (GUM) issue that occurs in almost all gradient surgery methods but has barely been discussed before. The GUM issue comes from the gradient manipulation of modern optimizers, which may destroy the conflict-free nature of the input gradient provided by the gradient surgery methods. The authors carefully analyze the GUM in PINNs training and propose a gradient update alignment to solve the GUM. The analysis is intensive, the experiment is solid, and the result is quite promising.

Critical strengths
  1. The manuscript comes with a detailed mathematical analysis. The reviewer particularly enjoyed Sec 3.3, where the authors carefully discuss which types of optimizer manipulations will break the conflict-freeness of the input gradient.
  2. The experimental setup is solid, with many ablations that consider different perspectives on the problem. The reviewer particularly liked Section 4.5. Although adding additional ablations would be helpful (e.g., test more values of ρ\rho instead of only 0 and 1), the current discussion already provides a good analysis of the state alignment method.
Critical weaknesses

The specific design of the state alignment is artificial and less motivated:

  • The core of GUM is actually a trade-off between conflict-freeness and nice features provided by the optimizer. If we want to fully respect the conflict-freeness of the gradient surgery method, we can use the vanilla SGD optimizer. If we want to utilize the nice feature provided by other modern optimizers, e.g., Adam, we have to sacrifice the conflict-freeness to allow the optimizer to modify the gradient via the optimizer states.
  • From this point of view, the state alignment method is actually the core of the proposed GUA method, as it aims to maintain the nice feature of the optimizers as much as possible while maintaining the conflict-freeness. If this part does not work well, the overall method will be close to the vanilla SGD optimizer+gradient surgery method.
  • Unfortunately, the current design of the state alignment method is not "elegant", i.e., it is not based on any mathematical motivation but explicit manual correction. Moreover, the coefficients used for optimizer-state alignment are hyperparameters that need to be finetuned for different experiments (Table 5). If we need to fine-tune hyperparameters to make the optimization work, we can simply add weights to each loss term and find the best weights. This is much cheaper since we don't need to use additional backpropagation to get loss-specific gradients for gradient surgery methods.
Questions for authors
  • Can the author further explain the RgR_g in Eq.3?
    • It seems that the authors want to measure the conflicts between the loss-specific gradients, but it is not clear why i<ji<j is required. Shouldn't it be i≠ji\ne j?
    • The reviewer will argue that RgR_g does not make too much sense. Even ⟨gi,gj⟩<0\langle g_i,g_j \rangle <0, it does not mean that there is conflict in the optimization. This is because we will never use a single gig_i or gjg_j to update the network. For example, if ⟨gi,gj⟩<0\langle g_i,g_j\rangle<0 but ∣gi∣=∣gj∣><0|g_i|=|g_j|><0, the update direction without any gradient surgery method, i.e., gi+gjg_i+g_j is conflict-free to both gig_i and gjg_j. If the author wants a value to represent the "difficult/conflict" of a multi-loss optimization problem, ⟨∑igi,gj⟩\langle \sum_i g_i,g_j \rangle is a better value to monitor.
  • Very recently, a concurrent preprint has also discussed the impact of the optimizer on conflict-free methods: Cautious Conflict-Free Inverse Gradient Methods (https://arxiv.org/abs/2610.31855). Could the author give some comments on the differences between GUM and this Cautious ConFIG?
  • Other minor issues/questions:
    • The last line on page 1 (Liu et al., 2025) is cited twice for the same sentence.
    • In line 209, do the authors mean "does not hold" instead of "need not hold"?
    • In Eq.6, the definition of $\widetilde{m} $ and $\widetilde{v} $ never occurs in the main content.
Reviewer_8xYBOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies Gradient Update Mismatch (GUM). A direction constructed to be conflict-free may become conflicting after an optimizer transformation. The authors characterize cone-preservation conditions and propose Gradient Update Alignment (GUA). GUA projects the optimizer proposal onto the current common-descent cone. It also softly aligns stored optimizer moments with the applied update. The PINN experiments largely follow the training and loss-grouping setup of ConFIG [11]. They cover six PDE benchmarks, 2 and 3 loss formulations, several gradient-manipulation methods, and multiple optimizers. Further experiments examine one-step changes, optimizer-state alignment, and task scaling on CelebA. The authors report fewer update conflicts and lower PINN solution errors. The state-alignment ablation suggests that soft correction of optimizer memory is an important part of the reported improvements.

Critical strengths

I see two positive aspects, although neither resolves my main concerns.


1. A useful diagnostic distinction between constructed and applied directions

The paper highlights a gap between guarantees on a constructed direction and the update applied by an optimizer. The cone-preservation analysis makes this distinction precise. GUM is an interesting diagnostic.


2. Soft optimizer-state alignment as a potential optimization correction

The adjustment of stored optimizer moments is the most promising part of GUA. Figure 5 suggests that soft alignment can work better than projection alone or aggressive state replacement. This mechanism may offer a useful way to correct optimizer dynamics. It is also the main reason I found something informative in the paper.


Critical weaknesses

The following are major concerns. They underlie my soundness assessment of 1.


1. GUM is demonstrated geometrically, but its relevance to PINN optimization is not established

The paper moves from identifying a violation of instantaneous common descent to treating it as a harmful optimizer update. These are different claims. Proposition 1 characterizes preservation of the entire cone. It does not show that violating this condition worsens optimization. Proposition 2 gives a sufficiently-small-step comparison. It does not establish better convergence or final PDE accuracy. Projection onto the cone boundary can still increase an individual loss through higher-order terms.

This distinction matters for ill-conditioned PINNs [1, 2]. Momentum and preconditioning use historical gradient information and coordinate-dependent scaling. They can filter or rescale a noisy first-order signal. Testing their updates only against the instantaneous first-order cone may discard useful conditioning information.

A simple example illustrates the issue. Let

L_2(x,y)=x^2(x-1)^2+5000y^2.$$ At $\theta=(0.75,0.001)$, the gradients are $g_1=(1.5,10)$ and $g_2=(-0.1875,10)$. Their sum $a=(1.3125,20)$ belongs to the common-descent cone. The exact Hessian of $L_1+L_2$ is $H=\operatorname{diag}(1.75,20000)\succ0$. Its Newton proposal is $u=H^{-1}a=(0.75,0.001)$. Yet $g_2^\top u=-0.130625<0$, so this is GUM under Definition 1. Nevertheless, $\theta-u=(0,0)$ makes both losses exactly zero in one step. This nonconvex, finite-step example does not contradict Proposition 2. It shows that GUM does not necessarily identify a harmful curvature-aware update. The stochastic setting raises another concern. Appendix B.3 resamples collocation points every epoch. GUA therefore enforces agreement with sampled gradients. The constraint $\widehat g_{i,t}^\top p_t\ge0$ does not imply $\nabla L_i(\theta_t)^\top p_t\ge0$ for the continuous-domain objective. MoCo [3] and MoDo [4] explain why stochastic gradient correction and conflict avoidance require care. I wonder whether soft state alignment partly stabilizes noisy moment estimates rather than correcting a genuinely harmful GUM event. This is a plausible alternative mechanism, not a demonstrated property of GUA. The experiments do not settle the issue. Table 1 reports $R_u$ for nine optimizers, but Appendix C.5 reports solution errors for only five. SGD, momentum SGD, SophiaG, and AdaHessian are omitted. SGD has $R_u=0$ by construction, yet its accuracy is not reported. SGD need not perform best, but it is an essential control. In Schrödinger with two losses, RMSProp has $R_u=1.8\%$, compared with $23.7\%$ for Adam, while Figure 10 shows much worse accuracy for RMSProp. The reported $R_u$ therefore does not consistently track optimizer performance. Q3 provides limited causal evidence. It examines selected early, middle, and late states only when $u_t$ is conflicting and $p_t\ne0$. It also evaluates changes on the same sampled batch. The reported metric is the maximum relative one-step loss change, whereas Proposition 2 concerns absolute changes. Relative normalization can change the worst-objective index and favor components with small current losses. The norm-matched comparison controls Euclidean magnitude, not magnitude in the adaptive metric. These conditional results do not establish better population-loss behavior or long-term optimization. Finally, the effect of post-optimizer projection is not separated from other mechanisms. For methods whose constructed directions already conflict, GUA also corrects ordinary gradient conflict. Related cone-based methods such as DCGD [5] and HARMONIC [6] are therefore relevant to attribution. For ConFIG [11], the state-alignment ablation makes changes to optimizer memory especially important. Modifying the second moment changes subsequent preconditioning. The existing momentum and state-decay controls do not fully isolate this effect. The relation to single-objective update-alignment work, particularly Cautious Optimizers [7] and MGUP [8], also needs clearer positioning. The paper may be stronger if it focuses on soft state correction rather than treating GUM elimination as the central explanation. --- ### 2. Oracle-assisted main evaluation prevents a reliable assessment of practical forward-PDE accuracy Appendix B.3 selects the checkpoint with the best validation performance. The released implementation uses reference-solution MSE for this selection and restores the corresponding weights. In the examined PDE implementations, validation and final evaluation also reuse the same reference grids. This directly affects the main accuracy results. A forward PDE solver generally does not know the solution it is trying to compute. It cannot use this information to select checkpoints. The fact that ConFIG [11] used a similar reporting convention does not resolve the issue. PINNacle [9] reports the last recorded evaluation, while SPINN [10] explicitly selects by training loss. These demonstrate feasible non-oracle alternatives. Applying the same oracle selection rule to both baseline and proposed methods also does not ensure equal benefits. Hyperparameter selection must follow the same principle. Appendix C.2 selects one $(\rho_m,\rho_v)$ pair per PDE based on validation performance. The exact selection objective, aggregation across settings, and search records are unclear. Figure 10 also does not specify whether the coefficients were transferred or retuned for each optimizer. Merely reporting final-epoch errors with previously reference-tuned coefficients would not resolve the oracle dependency. Both checkpoint selection and hyperparameter optimization must be repeated without access to the target reference solution. The evaluation further uses only two or three aggregated objectives. Coupled momentum and continuity residuals are grouped into a single PDE loss. The common-descent guarantee therefore applies to these groups, not necessarily to each physical equation. Appendix B.1 also changes relative loss weights between the two- and three-loss formulations for several PDEs. These comparisons do not isolate the effect of objective cardinality. ---
Questions for authors

1. Main evaluation and hyperparameter selection

What exact validation data and objective were used for checkpoints and PDE-wise coefficients? How is the stated test-set separation reconciled with the shared reference grids? Please rerun Figure 3 with non-oracle checkpoint selection and non-oracle hyperparameter optimization under comparable search budgets. Final-epoch reporting with reference-tuned coefficients is insufficient. Were Figure 10's coefficients transferred from Adam or retuned for each optimizer?


2. Evidence that GUM harms optimization

Please report final-checkpoint errors, convergence curves, runtimes, and configurations for all nine Table 1 optimizers. Does RuR_u, or the magnitude of its violations, predict degraded solution accuracy after controlling for optimizer-specific trajectories and preconditioning?


3. Projection, state correction, and existing methods

Under comparable tuning budgets, how do projection alone, first-moment-only correction, second-moment-only correction, and their combination perform? Could soft state correction improve stochastic optimizer statistics independently of GUM elimination? Please clarify the distinction from DCGD [5], HARMONIC [6], Cautious Optimizers [7], and MGUP [8]. A comparison with a SOAP-based PINN optimizer [2] would also be important for assessing the proposed method's effectiveness.


4. Q3 protocol and measurements

Which trajectory supplies the diagnostic states? What are the sampled steps, retention rates, and Figure 4 aggregation rules? Please report signed absolute and relative worst-objective changes, the frequency of simultaneous non-increase, and results on independent collocation samples. The raw records and diagnostic code are also needed.


References

[1] Rathore et al., "Challenges in Training PINNs: A Loss Landscape Perspective." ICML 2024.

[2] Wang et al., "Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective." NeurIPS 2025.

[3] Fernando et al., "Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Approach." ICLR 2023.

[4] Chen et al., "Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance." NeurIPS 2023.

[5] Hwang et al., "Dual Cone Gradient Descent for Training Physics-Informed Neural Networks." NeurIPS 2024.

[6] Bu et al., "Harmonized Cone for Feasible and Non-conflict Directions in Training Physics-Informed Neural Networks." ICLR 2026.

[7] Liang et al., "Cautious Optimizers: Improving Training with One Line of Code." ICLR 2026.

[8] Chang et al., "MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization." NeurIPS 2025.

[9] Hao et al., "PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs." NeurIPS 2024, Datasets and Benchmarks Track.

[10] Cho et al., "Separable Physics-Informed Neural Networks." NeurIPS 2023.

[11] Liu et al., "ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks." ICLR 2025.

Reviewer_KRAYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

A paper in the line of gradient surgery methods. Gradient surgery methods combine gradients gi,tg_{i,t} that come from different loss terms in an aggregate direction, ata_t, that does not not conflict to the gradient components that come from the different losses. The main observation of the paper is that while gradient surgery does remove conflicts its output ata_t is passed to optimizers; these optimizers have states/memory which they use to transform the conflict free direction ata_t to a new update direction, utu_t, that might not be conflict free. The paper assumes/models the optimizer transformation as an affine transformation, TT, and gives the conditions under which utu_t is conflict free (the linear component of TT should let all g_i gradients in the conflict free cone).

It proposes to correct the utu_t by projecting it back to the conflict free cone (direction alignment); it also add/explores a (soft) state correction (optimizer state alignement) that moves the optimizer state towards targets reconstructed form new aligned direction.

The experiments (on PINNs datasets/methods, across a set of conflict resolution methods ) explore whether:

  • the corrections are needed, i.e. does the original utu_t reintroduce conflicts? Answered in the positive (otherwise the paper would not exist of course)
  • do the alignment rules result in conflict free updates and performance improvement? no conlict resolutions exist after the correction, nrmal since we project in the conflict free cone. The PINN error with the correction is reduced in what seem to be statistical significant reductions.
  • what are the benefits of direction alignment? most of the benefit comes from the change of the direction, a smaller gain comes from the norm change.
  • is state alignment needed? soft state alignment brings benefits, significant in a good number of the datasets.
  • computational overhead? as expected it does have an overhead that depends on the problem and the number of loss components.
  • scalability with loss components? here the benchmark is a multi-task problem defined on CelebA and the evaluation measure becomes the average F1 measure. The results for the particular benchmark do not seem to give a significant improvement if one does the correction.
Critical strengths
  • very well written, a pleasure to read.
  • simple but good idea, supported by a detailed empirical analysis.
Critical weaknesses
  • The method demonstrates performance improvements in the PINN setting, but not in the CelebA evaluation in which the goal is to check scalability with the number of tasks.
Questions for authors

The non-significant performance in the CelebA task is not discussed. While the correction seems to bring performance improvements on the PINNs setting these do not transfer to the CelebA setting. The method as it is, is directly applicable to the multi-task setting. I would have welcomed a discussion here on why it does not tranfer there, and additional experiments with other multi-task benchmarks. The goal is not necessarily to show that it works there, but rather to try to gain an understanding on what are the structural differences of the PINNs setting vs the multi-task setting to which the different behaviour could be attributed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.