acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Backpropagation for Stable Physics-Informed Neural Networks

Abstract

Physics-informed neural networks (PINNs) differentiate losses containing spatial or temporal derivatives of network outputs, making backpropagation expensive and potentially unstable. We study a semi-gradient path that preserves residual values while replacing the residual Jacobian with the network-output Jacobian. This reduces update cost and can improve numerical stability, but prolonged use may cause oscillations and limit accuracy. We propose Prediction-guided Adaptive Semi-gradient Switching (PASS), which starts with semi-gradients and switches once to full gradients when successive prediction changes on a fixed probe set lose positive alignment. The criterion requires only output evaluations and no reference solution. Experiments demonstrate improved accuracy across optimizers, architectures, and PINN training techniques, at computational cost comparable to full-gradient training.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Semi-gradient, PINN, physics, backpropagation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hsw7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates the backward pass of PINNs. Computing PDE residuals requires forward evaluation of coordinate derivatives, and parameter updates require differentiating these coordinate derivatives with respect to parameters via chain rule expansion. This process is both computationally expensive and potentially unstable. The author proposes substituting the residual Jacobian with the network output Jacobian while keeping the residual forward value unchanged, and introduces a heuristic switching criterion that records output displacements on a fixed probe set every KK updates. The methods are evaluated on three PDEs.

Critical strengths

The paper cleanly dissects the backward path into coordinate derivatives and parameter sensitivity. It effectively illustrates the scale discrepancy between full and semi-gradients.

Critical weaknesses

The contribution is limited. The main technique semi-gradient backward path is largely drawn from existing literature (Cao & Zhang, 2026). The primary contribution is a heuristic switching criterion without theoretical convergence guarantees. This means that the work is more as an effective engineering trick than a fundamental methodological breakthrough. The experimental gains are also very limited, e.g. 0.924e-3 to 0.808e-3 in Table 4 for the reaction-diffusion equation.

Questions for authors
  1. Can the authors provide a compute-matched comparison against optimized fixed switching baselines along with the distribution of actual PASS switching times to substantiate the advantage of adaptive switching?

  2. How sensitive is the method to hyperparameters such as the observation interval KK, probe set size/placement, and the cosine threshold definition? What is the quantitative impact of false switches or missed switches on final accuracy?

  3. Why does PASS never trigger a switch in the biharmonic experiments? Does this indicate that the heuristic is insensitive to certain PDE families, and should there be a fallback trigger (e.g., based on gradient norm ratios) if a budget expires without switching?

  4. How does it behaves on exsiting PINN and its variants failed-training cases? Can it successfully solve a PDE case where existing PINN variants failed to train?

Reviewer_q8e3OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Prediction-guided Adaptive Semi-gradient Switching (PASS), an adaptive training algorithm for physics-informed neural networks (PINNs) where the network is initially trained by "semi-gradients" before switching to the full residual training of PINNs to accelerate training and reduce the computational costs. During the semi-gradient training phase, the model parameters are only updated via the Jacobians of a portion of the residual loss that is directly controlled by the network outputs uu and not the Jacobians of the other residual terms. The authors demonstrate that this approximate training accelerates the convergence, but eventually results in oscillatory behaviour that requires a full residual training phase in order to fully converge. The transition to the full training is decided by measuring the change of direction in network output updates across training epochs. PASS is evaluated for reaction-diffusion and Burgers' equations and is shown to be compatible with different architectures and various PINN training methods.

Critical strengths
  1. PASS is easy to implement, does not induce additional computational costs, and effectively improves the stability and convergence of PINNs.
  2. For linear PDEs, PASS objective aligns with the full residual training with improved convergence rate.
Critical weaknesses
  1. The novelty and contribution of PASS are very limited. PASS's main contribution is replacing the weighted sum of full and semi-gradients (SGR) with a 2-phase semi- and full-training approach.
  2. Although the authors experiment with two nonlinear equations, the justification for semi-gradients does not extend to nonlinear PDEs. In my view, semi-gradients remain an empirical method for finding an "initialization" where the full residual is well-behaved. This is a valuable contribution but requires extensive experiments over more nonlinear and challenging PDEs for validation.
  3. Original PINNs and many subsequent works used L-BFGS or a 2-phase Adam and L-BFGS training for better convergence. The authors train baselines using Adam and SGD only.
  4. The computational savings from PASS might be overstated. PASS still requires full residual training, and thus does not provide meaningful memory savings. Furthermore, the wall-time savings from PASS compared to full training are not significant. Meanwhile, the authors do not report when (at what epoch) the transition from semi-gradient to full training happens in each experiment, which raises the question of whether full gradients comprise the majority of the training iterations. In Figure 3, Semi and PASS quickly diverge from each other, which, based on my understanding, indicates a fast transition from semi to full training.
Questions for authors

Please see the weaknesses. Also,

  1. How do the other residual terms (r−ur-u) and their gradients (magnitude) change during the semi-training (especially for nonlinear PDEs)? Are they consistent with the error and semi-gradients until the transition?
  2. Is there a specific reason why PASS results are not reported for the biharmonic equation? I udenrstand the semi training alone already performs well for biharmonic, but reporting PASS results at least in the Appendix would be appreciated.
  3. (W.4) What portion of the training iterations in each experiment are dedicated to semi and full training?
Reviewer_z1wnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the use of semi-gradient updates for training physics-informed neural networks (PINNs) and introduces PASS, an adaptive strategy that switches from semi-gradients to standard residual gradients during optimization.

Standard PINN training requires differentiating the PDE residual with respect to network parameters, including mixed derivatives involving spatial or temporal coordinates. Building on previous gradient-residual methods, the authors consider a semi-gradient that replaces the residual Jacobian with the network-output Jacobian while preserving the numerical value of the residual.

The paper discusses the computational and optimization properties of these two directions. In particular, for a symmetric positive-definite linear system optimized directly in solution space, the semi-gradient avoids the squared condition number associated with residual least-squares minimization.

However, prolonged semi-gradient optimization may produce oscillations and limit final accuracy. To address this, PASS monitors predictions on a fixed set of points and switches permanently to full gradients when two successive prediction displacements have nonpositive cosine similarity.

Experiments on biharmonic, Burgers, and reaction–diffusion equations demonstrate improved numerical stability, reduced computational cost during the semi-gradient phase, and improved final accuracy across several training configurations.

Critical strengths
  1. A simple and computationally inexpensive strategy for improving PINN training.

The proposed approach combines the computational advantages of semi-gradient updates with the refinement capabilities of conventional residual gradients. PASS requires only occasional evaluations of the predicted solution and a one-time modification of the backward computation. It does not require reference solutions or repeated evaluations of both gradient directions, making it relatively easy to integrate into existing PINN implementations.

The biharmonic experiment provides convincing evidence that semi-gradient backpropagation can substantially reduce computational cost and improve numerical stability over the tested learning-rate range.

  1. Interesting experimental evidence for combining semi-gradient and full-gradient optimization.

The experiments illustrate the complementary properties of the two update directions. In particular, semi-gradient optimization can provide rapid initial improvements, whereas prolonged use may produce oscillations or limit final accuracy.

The Burgers experiment demonstrates that switching to full gradients can improve final accuracy, while the reaction–diffusion experiments suggest that the strategy can be beneficial across different architectures, optimizers, and PINN training techniques. These results support the general idea that changing the backward computation during training can improve optimization.

Critical weaknesses
  1. The proposed switching criterion lacks a convincing connection to the optimization mechanism it is intended to detect.

The main methodological contribution of the paper is the adaptive switching criterion based on the cosine similarity of successive prediction displacements. However, this quantity does not directly characterize whether the semi-gradient remains a descent direction for the original residual loss, nor whether switching to the full gradient would improve convergence.

As the authors correctly observe in Equation (7), the semi-gradient is a descent direction for the original objective only when

⟨gF,gS⟩>0.\langle g_F,g_S\rangle > 0.

However, the proposed criterion instead monitors the alignment of successive changes in the predicted solution, which is a different geometric quantity. In particular, a negative prediction-displacement cosine may occur along an otherwise convergent trajectory, while a positive cosine may persist even when the semi-gradient is not a descent direction.

A simple example illustrates this limitation. Consider the scalar equation u=0u=0, parameterized by uθ=θu_\theta=\theta, with residual rθ=−θr_\theta=-\theta. The full gradient is gF=2θg_F=2\theta, whereas the semi-gradient is gS=−2θg_S=-2\theta. Consequently, semi-gradient descent with a positive constant learning rate diverges geometrically, although successive prediction displacements remain positively aligned. In this example, the PASS criterion would never trigger a switch.

This observation is closely related to the residual-sign dependence acknowledged by the authors in Section 3.2: replacing rθr_\theta by −rθ-r_\theta leaves the least-squares objective unchanged but reverses the semi-gradient PDE contribution.

Although the authors explicitly recognize that their criterion is heuristic, the paper does not sufficiently establish why prediction-displacement alignment should reliably indicate when semi-gradient optimization becomes inappropriate. Since this criterion constitutes the main methodological novelty, I consider this an important limitation.

  1. The experiments do not sufficiently isolate the benefit of the adaptive switching rule from alternative optimization strategies.

The comparison with fixed switching schedules is useful, but the tested schedules are relatively coarse. In the Burgers experiment, the authors consider fixed switches at 10%, 25%, 50%, and 75% of training, with only the last remaining numerically stable. However, the paper does not report the distribution of actual PASS switching times.

A particularly informative comparison would involve fixed switching schedules near the times selected by PASS, including a fixed switch at the median PASS switching step. This would help determine whether the improvement comes specifically from the proposed prediction-based criterion or more generally from a suitable semi-gradient warm-up.

Furthermore, the paper motivates semi-gradients partly through the computational cost of differentiating the PDE residual. However, it does not compare against alternatives based on freezing or periodically refreshing the residual Jacobian. For example, one could consider an update direction of the form

Jr(θˉ)⊤rθ,J_r(\bar\theta)^\top r_\theta,

where θˉ\bar\theta is a reference parameter value, fixed initially or updated periodically. Such approaches could also reduce the cost of repeatedly differentiating the PDE residual, although their memory requirements and Jacobian-refresh costs would need to be accounted for.

Comparisons against these alternatives, as well as adaptive switching criteria based on residual stagnation or prediction-update magnitudes, would help isolate the specific contribution of PASS.

Finally, the experiments use only three random seeds per configuration, and some improvements over stabilized gradient-residual training are relatively small compared with the reported variability. Since the two backward paths can also have substantially different gradient magnitudes, additional comparisons with appropriately stabilized or gradient-normalized full-gradient optimization would strengthen the conclusions.

Additional minor remark: Consistency of the semi-gradient formulation for PDE systems.

The proposed semi-gradient gS=Ju⊤rθg_S=J_u^\top r_\theta implicitly identifies the space of PDE residuals with the space of network outputs. While this is consistent for the scalar equations considered in the paper, it is not necessarily well-defined for more general PDE systems.

In particular, when uθ:Ω→Rmu_\theta:\Omega\to\mathbb R^m and rθ:Ω→Rqr_\theta:\Omega\to\mathbb R^q with m≠qm\neq q, the matrix product Ju⊤rθJ_u^\top r_\theta is dimensionally inconsistent.

A more general formulation would require specifying a mapping between the residual and solution spaces, potentially changing the resulting optimization direction and its conditioning properties. Although the authors explicitly restrict their analysis to scalar PDEs, this raises a question about the consistency and applicability of the proposed semi-gradient construction to more general physics-informed learning problems.

Questions for authors
  1. [Main concern: switching mechanism] Have the authors investigated the relationship between the prediction-displacement cosine and the alignment between the full and semi-gradients, particularly the sign of ⟨gF,gS⟩\langle g_F,g_S\rangle? Does the proposed criterion reliably detect when semi-gradient updates cease to decrease the original objective? How sensitive is PASS to equivalent PDE formulations involving a reversal of the residual sign?

  2. [Main concern: attribution of the adaptive rule] Could the authors report the actual switching times across seeds and experiments, and compare PASS against fixed switches near these times? In particular, how does PASS compare with a fixed switch at the median PASS switching step, or with inexpensive alternative indicators such as residual stagnation or prediction-update magnitudes? How sensitive are the results to the observation interval KK and the choice of probe points?

  3. [Main concern: alternative optimization baselines] Could the authors compare PASS with alternative strategies that avoid repeated residual-Jacobian differentiation, such as frozen or periodically refreshed residual Jacobians? Comparisons under matched computational budgets, accounting for Jacobian storage and refresh costs, would help establish whether the observed benefits are specific to the semi-gradient direction. Additionally, would the advantages persist against full-gradient optimization with appropriate gradient normalization or other standard stabilization mechanisms?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.