acceptodds
Under review as a conference paper at ICLR 2027

Regressive-Dual PINN: Faster and More Reliable Training for Long-Horizon PDEs.

Abstract

Physics-informed neural networks (PINNs) use reverse-mode automatic differentiation (AD) to solve partial differential equations (PDEs), and are expected to run fast on advanced AI hardware. In practice, their speed is far below expectations, and reliability over long time horizons is poor, often converging to trivial solutions. Prior fixes, such as finite-difference (FD) schemes and window-based training, either sacrifice accuracy or ignore the relation between successive PINN models. To address these shortcomings, we propose Regressive Dual-PINN (RD-PINN), comprising two components: Dual-FD and regressive initialization. Dual-FD computes exact first-order derivatives via dual numbers and second-order derivatives via FD, significantly reducing pure-FD's approximation error while avoiding AD's costly nested backward passes. Regressive initialization exploits the trajectory of models trained across earlier time windows, using regression to initialize a new window's parameters. Across seven PDE problems and 10 architectures, Dual-FD shows mixed speedups on one-dimensional problems. In 24 paired two-dimensional comparisons, its median speedup is 2.42× and its maximum is 9.68×; peak memory is lower in 21 of the 22 comparisons. Regressive initialization's advantage emerged on five unsteady PDE problems with extended horizons: only RD-PINN consistently sustained correct long-horizon behavior without reference data, while four published window-based and the short-horizon methods degraded as dimensionality and horizon length grew.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Physics-informed neural networks, Automatic differentiation, Dual numbers, Long-horizon PDEs, Time-window training, Trajectory-based initialization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FpkHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes regressive dual-PINN to improve the speed and reliability of physics-informed neural networks for solving PDEs over long time horizons. It introduces two components: Dual-FD, which combines dual numbers for first-order derivatives with finite differences for second-order derivatives, and regressive initialization, which uses historical neural network weights to initialize subsequent time windows. The experiments show promising speedups, reduced memory usage, and improved long-horizon predictions.

Critical strengths
  • Dual-FD reduces the computational cost of derivative calculations while maintaining accuracy.
  • The method achieves speedups, particularly for two-dimensional PDEs, and reduces GPU memory consumption.
  • Regressive initialization preserves important physical behavior over longer time intervals, addressing a known limitation of traditional PINNs.
Critical weaknesses
  • Dual numbers, finite differences, and parameter extrapolation are established techniques. The paper does not sufficiently clarify its methodological novelty relative to related approaches.
  • The paper lacks direct experimental comparisons with relevant differentiation methods such as DOF, CAN-PINN, and STDE, and temporal approaches such as E-DNN.
  • The paper does not clearly isolate the contribution of regressive initialization from Dual-FD. Therefore, it is difficult to determine which component dictates the long-horizon improvements.
  • Although numerical errors are discussed, the paper does not deeply investigates how approximation errors and prediction errors accumulate across consecutive time windows.
  • The sensitivity of regressive initialization to window size and historical parameter changes is unclear. Additionally, differences in compilation settings between Dual-FD and AD may affect the reported speedups.
Questions for authors
  • What is the main methodological novelty of Dual-FD compared with existing hybrid differentiation techniques, and how does regressive initialization differ from conventional parameter extrapolation?
  • Can the authors provide comparisons with recent differentiation methods such as DOF and CAN-PINN, as well as temporal approaches such as E-DNN, to better demonstrate the advantages of RD-PINN?
  • Can the authors evaluate Dual-FD and regressive initialization independently, including comparisons against simple weight copying and extrapolation, to quantify each component's contribution?
  • How do numerical approximation errors and prediction errors propagate across time windows? Does regressive initialization reduce or potentially amplify accumulated errors over longer horizons?
  • How sensitive is the method to window size and the quality of historical parameter estimates? Can the authors also report speed comparisons under matched compilation settings?
11 comments
AuthorsOpenReview ↗

1. Response to W1 (Part 1) and Q1 (Part 1): Dual-FD versus existing differentiation methods

We sincerely thank the reviewer for the constructive feedback. We appreciate the recognition that Dual-FD reduces the cost of derivative calculations while maintaining accuracy and lowers GPU memory consumption, with speedups in particular on two-dimensional PDEs, and that regressive initialization preserves important physical behaviour over longer time intervals, addressing a known limitation of traditional PINNs. Below, we carefully address the reviewer's concerns and provide further clarifications and additional evidence to strengthen our work.

We clarify the distinction between Dual-FD and existing differentiation methods through their computational mechanisms and direct comparisons.

Dual-FD combines two established techniques in a simple way. To our knowledge, no prior PINN method combines first-order dual-number differentiation with a single centered difference of exact first derivatives to compute second-order PDE residuals. Dual-FD avoids nested reverse-mode AD and explicit second-order derivative propagation while keeping first derivatives exact. This simple combination reduces differentiation costs without changing the network architecture or training procedure, as demonstrated across seven PDEs and ten architectures. We regard this simplicity as an advantage, as the operator is easy to adopt in existing PINN code, and the gains come from the combination rather than from additional machinery.

Higher-order forward AD extends first-order forward differentiation using nested dual numbers (forward-over-forward), Taylor-mode AD, or Forward Laplacian propagation. The nested computation of second- and higher-order derivatives slows down the whole process.

Dual-FD takes a different approach: it uses dual numbers to compute only the first-order derivatives and approximates second derivatives through centered finite differences:

uxx(x)≈uxdual(x+h)−uxdual(x−h)2h.u_{xx}(x)\approx\frac{u_x^{\mathrm{dual}}(x+h)-u_x^{\mathrm{dual}}(x-h)}{2h}.

Thus, no second-order derivatives are propagated through the network. For smooth activation networks, direct second-order AD involves σ′′\sigma'' in the PDE residual and requires derivatives up to σ′′′\sigma''' during parameter backpropagation. Dual-FD instead involves only σ\sigma and σ′\sigma' in the residual, requiring derivatives up to σ′′\sigma'' during training. This reduces higher-order differentiation costs.

AuthorsOpenReview ↗

2. Response to W1 (Part 1) (Cont.): Comparison with forward-mode, higher-order and hybrid methods

Forward-mode and higher-order differentiation

SPINN [1] differs from Dual-FD in two respects. First, it applies the dual pass twice (forward over forward) to obtain second derivatives, which places σ′′\sigma'' in the residual, whereas Dual-FD applies it once. Second, it requires a separable architecture with one subnetwork per input coordinate, whereas Dual-FD works on any network. On SPINN's own 3D Helmholtz case, replacing only its differentiation with Dual-FD makes each training step 4.2× faster and reaches the error thresholds 2.9–4.8× sooner, with comparable final errors.

Taylor-mode AD [2] extends forward differentiation to higher orders by propagating Taylor derivative information through network layers. DOF [3] uses Forward Laplacian propagation to compute the gradient and Laplacian directly without constructing the full Hessian. STDE [4] estimates differential operators using randomly sampled Taylor-mode directions, reducing high-dimensional computation at the cost of sampling variance. Cao et al. [5] build an explicit forward-propagation kernel for each PDE residual, which propagates only the derivative states that residual needs. The kernel is specific to the residual and the network, so a new residual needs a new kernel.

Dual-FD requires only PDE-independent first-order forward differentiation rules, so new residuals need no new derivative kernels. On Kovasznay and Taylor–Green, Dual-FD is 1.6–1.8× faster per training step than Taylor-mode AD, 1.9–2.3× faster than DOF, and approximately 3.8–3.9× faster than nested AD. Since STDE is designed for high-dimensional problems, we evaluated both methods on a dd-dimensional Poisson equation. Dual-FD outperforms STDE in per-step runtime up to d=8d=8. For d=3d=3–77, Dual-FD achieves an average speedup of 1.26×, with a maximum of 1.83× at d=3d=3. STDE is faster than Dual-FD at the tested dimensions above d=8d=8. On Burgers 1D, the only case shared with Cao et al., Dual-FD is 1.4× faster than their released code on the same GPU at the same final loss.

Hybrid differentiation

CAN-PINN [6] also combines AD and finite differences. Its upwind approximation combines a FD of network values with an AD-based correction:

ux(x)≈u(x)−u(x−h)h⏟FD+uxAD(x)−uxAD(x−h)2⏟AD correction.u_x(x)\approx \underset{\mathrm{FD}}{\underbrace{\frac{u(x)-u(x-h)}{h}}} + \underset{\mathrm{AD\ correction}}{\underbrace{\frac{u_x^{\mathrm{AD}}(x)-u_x^{\mathrm{AD}}(x-h)}{2}}}.

Here, reverse-mode AD provides a correction to the FD estimate rather than serving as the final first-derivative operator. CAN-PINN computes second derivatives using centered differences of network values. This incorporates neighboring-point information into the residual to improve training with sparse collocation points.

Dual-FD instead obtains exact first derivatives using forward-mode AD and applies FD only to these derivatives, targeting the computational cost of second-order residuals. On Kovasznay and Taylor–Green, Dual-FD is 2.7–4× faster per step, reaches matched accuracy thresholds 1.8–2.3× sooner, and achieves 1.6–10× lower final errors. The methods differ in how they compute both first- and second-order derivatives. CAN-PINN computes first derivatives with a reverse pass over the coordinates, whereas Dual-FD obtains them in one forward dual pass. The second-order difference of network values in CAN-PINN carries a larger error than the single difference of exact first derivatives in Dual-FD.

Dual-FD combines first-order dual-number AD with centered differences of first derivatives. By avoiding nested reverse-mode AD and explicit second-order propagation, it reduces differentiation costs without changing the training procedure, and it requires neither a separable architecture, as SPINN does, nor residual-specific kernels, as Cao et al. do. The same operator applies to all ten architectures in the paper's Table 1.

[1] Cho, Junwoo, et al. "Separable physics-informed neural networks." Advances in Neural Information Processing Systems 36 (2023).

[2] Dangel, Felix, et al. "Collapsing Taylor mode automatic differentiation." Advances in Neural Information Processing Systems 38 (2025).

[3] Li, Ruichen, et al. "DOF: Accelerating high-order differential operators with forward propagation." arXiv preprint arXiv:2402.09730 (2024).

[4] Shi, Zekun, et al. "Stochastic Taylor derivative estimator: Efficient amortization for arbitrary differential operators." Advances in Neural Information Processing Systems 37 (2024).

[5] Cao, Wenbo, Zhe Lu, and Weiwei Zhang. "Verified residual-specific explicit derivative kernels for physics-informed learning and discretized PDE adjoints." arXiv preprint arXiv:2606.29702 (2026).

[6] Chiu, Pao-Hsiung, et al. "CAN-PINN: A fast physics-informed neural network based on coupled-automatic–numerical differentiation method." Computer Methods in Applied Mechanics and Engineering 395 (2022): 114909.

AuthorsOpenReview ↗

3. Response to W1 (Part 2) and Q1 (Part 2): Regressive initialization versus low-order extrapolation

We would like to clarify what regressive initialization is and how it differs from a low-order extrapolation.

The form. RD-PINN initializes window k+1k+1 as

θk+1(0)=θk+akdk+bkdk−1,dk=θk−θk−1,\theta_{k+1}^{(0)}=\theta_k+a_kd_k+b_kd_{k-1},\qquad d_k=\theta_k-\theta_{k-1},

and copying the previous window (ak=bk=0a_k=b_k=0) and linear extrapolation (ak=1a_k=1, bk=0b_k=0) are special cases of this form. The distinction lies in how the coefficients are obtained.

How the coefficients are obtained. Unlike conventional extrapolation with fixed coefficients, RD-PINN adaptively fits its coefficients using the parameter history accumulated across completed windows. Every completed window jj provides one observation consisting of the two preceding parameter increments, dj−1d_{j-1} and djd_j, and the subsequent increment dj+1d_{j+1}. The coefficients are the least-squares solution over all these observations (Sec. 3.3 and App. A.5 of the paper),

(ak,bk)=arg⁡min⁡a,b∑j=3k−1∥adj+bdj−1−dj+1∥22.(a_k,b_k)=\arg\min_{a,b}\sum_{j=3}^{k-1}\big\Vert a d_j+b d_{j-1}-d_{j+1}\big\Vert_2^2 .

Each completed window adds one observation to the fit, so the fit uses the whole history at negligible cost because of the cumulative update scheme (Eq. 14 and App. A.5 of the paper). The predicted update is bounded in size, ∥akdk+bkdk−1∥2≤τ∥dk∥2\Vert a_kd_k+b_kd_{k-1}\Vert_2\le\tau\Vert d_k\Vert_2 with τ=0.5\tau=0.5 by default, and the predicted parameters are only the starting point of the next window, which is then trained on the PDE residual as before. The two latest increments thus set the directions of the prediction, and the whole history sets how far to move along them.

To see what the regression changes, we rewrite it as

θk+1(0)=θk−2+(1+ak)Δθk−(ak−bk)Δθk−1,\theta_{k+1}^{(0)}=\theta_{k-2}+(1+a_k) \Delta\theta_k-(a_k-b_k) \Delta\theta_{k-1},

where θk−1=θk−2+Δθk−1\theta_{k-1}=\theta_{k-2}+\Delta\theta_{k-1} and θk=θk−2+Δθk\theta_k=\theta_{k-2}+\Delta\theta_k are the parameters reached by training. Ignoring the last term, θk+1(0)=θk−2+(1+ak)Δθk\theta_{k+1}^{(0)}=\theta_{k-2}+(1+a_k)\Delta\theta_k has a form similar to gradient descent, where 1+ak1+a_k is a learning parameter and Δθk\Delta\theta_k controls the direction, and the last term can be considered as a momentum term. In this form, copying fixes the learning parameter at 1 with no momentum term and linear extrapolation fixes it at 2 with a momentum coefficient of −1-1, whereas regression learns both from the completed windows. The regressed parameters are for initialization only and are trained on the residual loss, and a norm constraint restricts the difference between θk+1(0)\theta_{k+1}^{(0)} and θk\theta_k.

On Burgers 1D, regression starts at window 5 with one observation. Each completed window adds one observation and corrects the coefficients, and from window 9 onward they are nearly stable through window 40 (Table R1), because the early increments, whose norms are 4–6 times larger than the later ones, dominate the cumulative least-squares fit. The coefficients are thus learned from the run as it proceeds, which a rule fixed in advance cannot do.

Table R1. Fitted coefficients window by window, Burgers 1D

Window kk5678910203040
Completed windows used in the fit123456162636
aka_k0.380.350.310.300.290.290.290.290.29
bkb_k−0.01−0.05−0.05−0.05−0.06−0.06−0.05−0.05−0.04
Reviewer_K16kOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Regressive Dual-PINN (RD-PINN) to address two limitations of physics-informed neural networks (PINNs): expensive derivative computation and unreliable training for PDEs over long time horizons.

The method combines two techniques. Dual-FD uses forward-mode differentiation with dual numbers to compute first-order derivatives and finite differences of these derivatives to approximate second-order derivatives, avoiding expensive nested automatic differentiation. Regressive initialization divides the temporal domain into windows and uses least-squares regression over parameter changes from previously trained windows to initialize the next model.

Experiments across seven PDE problems and ten PINN architectures show substantial computational improvements, particularly for two-dimensional problems, with a reported median speedup of 2.42x and generally reduced GPU memory consumption. Long-horizon experiments on five PDE problems suggest that RD-PINN improves solution stability and preserves physical fluctuations more effectively than several existing temporal training approaches.

Critical strengths
  1. Dual-FD achieves meaningful computational improvements across a fairly broad range of PDE problems and PINN architectures. The results are particularly convincing for 2D problems with multiple second-order derivatives, where the speedups are consistent. The reduced memory consumption is also useful.
  2. The experiments are reasonably extensive, with comparisons across different architectures, PDEs, and differentiation schemes. The authors also examine where the computational savings come from, rather than just reporting overall speedups.
  3. The long-horizon experiments consider physical behavior beyond aggregate solution errors. This is important since PINNs may achieve relatively low errors while losing meaningful physical structures. The Taylor-Green and cylinder-flow results are interesting in this regard.
Critical weaknesses
  1. The novelty of the proposed techniques seems somewhat limited. Dual-FD combines established forward-mode differentiation and finite difference methods. While the combination is practically useful, the underlying idea is relatively straightforward. Similarly, regressive initialization is essentially a low-order extrapolation of neural network parameters across time windows.
  2. Related to the previous point, the paper could better position regressive initialization relative to existing work on sequential neural PDE solvers. For example, TENG [1], the sequential-in-time framework of Zhang et al. [2], and E-DNN [3] study related problems involving neural representations evolving over time. The distinction from these approaches, particularly in terms of parameter evolution and long-horizon reliability, deserves more discussion.
  3. I am also not fully convinced by the motivation for regressing directly in parameter space. Neural network parameters are not uniquely determined by the represented function, and their changes across windows do not necessarily follow a predictable trend. The method may work well empirically, but it is unclear when this extrapolation is expected to be reliable or when it could become unstable.
  4. More importantly, the contribution of regressive initialization is not sufficiently isolated. The long-horizon experiments compare RD-PINN with other temporal training frameworks, but these methods differ in several aspects besides initialization. A comparison using the same training framework, with and without regression, would be important.
  5. Related to point 4, simpler alternatives such as copying the previous window's parameters or using first-order extrapolation should also be considered. Without these comparisons, it is difficult to determine whether the improvement comes from the proposed regression over historical increments or simply from using a reasonable warm start.

[1] https://proceedings.mlr.press/v235/chen24ad.html

[2] https://epubs.siam.org/doi/abs/10.1137/24M1651241

[3] https://www.sciencedirect.com/science/article/pii/S0893608024009274

Questions for authors
  1. Could the authors clarify the main distinction between regressive initialization and existing sequential neural PDE methods? In particular, what makes extrapolation over historical parameter increments preferable to existing time-evolution strategies?
  2. Are there ablations comparing regressive initialization against simply copying the previous window's weights or using a simpler linear extrapolation, while keeping everything else fixed? This would help clarify how much of the long-horizon improvement actually comes from the proposed regression.
  3. How sensitive is regressive initialization to the window size and the history used for regression? Is there any evidence of cases where parameter extrapolation becomes unstable or provides a worse initialization?
9 comments
AuthorsOpenReview ↗

1. Response to W1 (Part 1): Dual-FD versus existing differentiation methods

We sincerely thank the reviewer for the constructive feedback. We appreciate the recognition that Dual-FD achieves meaningful and consistent computational improvements across a broad range of PDEs and architectures, with reduced memory consumption; that the experiments examine where the savings come from rather than only reporting speedups; and that the long-horizon experiments consider physical behaviour beyond aggregate errors, in particular on Taylor–Green and the cylinder flow. Below, we carefully address the reviewer's concerns and provide further clarifications and additional evidence to strengthen our work.

1. Response to W1 (Part 1): Dual-FD versus existing differentiation methods

We clarify the distinction between Dual-FD and existing differentiation methods through their computational mechanisms and direct comparisons.

Dual-FD combines two established techniques in a simple way. To our knowledge, no prior PINN method combines first-order dual-number differentiation with a single centered difference of exact first derivatives to compute second-order PDE residuals. Dual-FD avoids nested reverse-mode AD and explicit second-order derivative propagation while keeping first derivatives exact. This simple combination reduces differentiation costs without changing the network architecture or training procedure, as demonstrated across seven PDEs and ten architectures. We regard this simplicity as an advantage, as the operator is easy to adopt in existing PINN code, and the gains come from the combination rather than from additional machinery.

Higher-order forward AD extends first-order forward differentiation using nested dual numbers (forward-over-forward), Taylor-mode AD, or Forward Laplacian propagation. The nested computation of second- and higher-order derivatives slows down the whole process..

Dual-FD takes a different approach: it uses dual numbers to compute only the first-order derivatives and approximates second derivatives through centered finite differences:

uxx(x)≈uxdual(x+h)−uxdual(x−h)2h.u_{xx}(x)\approx\frac{u_x^{\mathrm{dual}}(x+h)-u_x^{\mathrm{dual}}(x-h)}{2h}.

Thus, no second-order derivatives are propagated through the network. For smooth activation networks, direct second-order AD involves σ′′\sigma'' in the PDE residual and requires derivatives up to σ′′′\sigma''' during parameter backpropagation. Dual-FD instead involves only σ\sigma and σ′\sigma' in the residual, requiring derivatives up to σ′′\sigma'' during training. This reduces higher-order differentiation costs.

AuthorsOpenReview ↗

2. Response to W1 (Part 1) (Cont.): Comparison with forward-mode, higher-order and hybrid methods

Forward-mode and higher-order differentiation

SPINN [1] differs from Dual-FD in two respects. First, it applies the dual pass twice (forward over forward) to obtain second derivatives, which places σ′′\sigma'' in the residual, whereas Dual-FD applies it once. Second, it requires a separable architecture with one subnetwork per input coordinate, whereas Dual-FD works on any network. On SPINN's own 3D Helmholtz case, replacing only its differentiation with Dual-FD makes each training step 4.2× faster and reaches the error thresholds 2.9–4.8× sooner, with comparable final errors.

Taylor-mode AD [2] extends forward differentiation to higher orders by propagating Taylor derivative information through network layers. DOF [3] uses Forward Laplacian propagation to compute the gradient and Laplacian directly without constructing the full Hessian. STDE [4] estimates differential operators using randomly sampled Taylor-mode directions, reducing high-dimensional computation at the cost of sampling variance. Cao et al. [5] build an explicit forward-propagation kernel for each PDE residual, which propagates only the derivative states that residual needs. The kernel is specific to the residual and the network, so a new residual needs a new kernel.

Dual-FD requires only PDE-independent first-order forward differentiation rules, so new residuals need no new derivative kernels. On Kovasznay and Taylor–Green, Dual-FD is 1.6–1.8× faster per training step than Taylor-mode AD, 1.9–2.3× faster than DOF, and approximately 3.8–3.9× faster than nested AD. Since STDE is designed for high-dimensional problems, we evaluated both methods on a dd-dimensional Poisson equation. Dual-FD outperforms STDE in per-step runtime up to d=8d=8. For d=3d=3–77, Dual-FD achieves an average speedup of 1.26×, with a maximum of 1.83× at d=3d=3. STDE is faster than Dual-FD at the tested dimensions above d=8d=8. On Burgers 1D, the only case shared with Cao et al., Dual-FD is 1.4× faster than their released code on the same GPU at the same final loss.

Hybrid differentiation. CAN-PINN [6] also combines AD and finite differences. Its upwind approximation combines a FD of network values with an AD-based correction:

ux(x)≈u(x)−u(x−h)h⏟FD+uxAD(x)−uxAD(x−h)2⏟AD correction.u_x(x)\approx \underset{\mathrm{FD}}{\underbrace{\frac{u(x)-u(x-h)}{h}}} + \underset{\mathrm{AD\ correction}}{\underbrace{\frac{u_x^{\mathrm{AD}}(x)-u_x^{\mathrm{AD}}(x-h)}{2}}}.

Here, reverse-mode AD provides a correction to the FD estimate rather than serving as the final first-derivative operator. CAN-PINN computes second derivatives using centered differences of network values. This incorporates neighboring-point information into the residual to improve training with sparse collocation points.

Dual-FD instead obtains exact first derivatives using forward-mode AD and applies FD only to these derivatives, targeting the computational cost of second-order residuals. On Kovasznay and Taylor–Green, Dual-FD is 2.7–4× faster per step, reaches matched accuracy thresholds 1.8–2.3× sooner, and achieves 1.6–10× lower final errors. The methods differ in how they compute both first- and second-order derivatives. CAN-PINN computes first derivatives with a reverse pass over the coordinates, whereas Dual-FD obtains them in one forward dual pass. The second-order difference of network values in CAN-PINN carries a larger error than the single difference of exact first derivatives in Dual-FD.

Dual-FD combines first-order dual-number AD with centered differences of first derivatives. By avoiding nested reverse-mode AD and explicit second-order propagation, it reduces differentiation costs without changing the training procedure, and it requires neither a separable architecture, as SPINN does, nor residual-specific kernels, as Cao et al. do. The same operator applies to all ten architectures in the paper's Table 1.

[1] Cho, Junwoo, et al. "Separable physics-informed neural networks." Advances in Neural Information Processing Systems 36 (2023).

[2] Dangel, Felix, et al. "Collapsing Taylor mode automatic differentiation." Advances in Neural Information Processing Systems 38 (2025).

[3] Li, Ruichen, et al. "DOF: Accelerating high-order differential operators with forward propagation." arXiv preprint arXiv:2402.09730 (2024).

[4] Shi, Zekun, et al. "Stochastic Taylor derivative estimator: Efficient amortization for arbitrary differential operators." Advances in Neural Information Processing Systems 37 (2024).

[5] Cao, Wenbo, Zhe Lu, and Weiwei Zhang. "Verified residual-specific explicit derivative kernels for physics-informed learning and discretized PDE adjoints." arXiv preprint arXiv:2606.29702 (2026).

[6] Chiu, Pao-Hsiung, et al. "CAN-PINN: A fast physics-informed neural network based on coupled-automatic–numerical differentiation method." Computer Methods in Applied Mechanics and Engineering 395 (2022): 114909.

AuthorsOpenReview ↗

3. Response to W1(Part 2): Regressive initialization versus low-order extrapolation

We would like to clarify how regressive initialization differs from low-order extrapolation.

RD-PINN initializes window k+1k+1 as

θk+1(0)=θk+akdk+bkdk−1,dk=θk−θk−1.\theta_{k+1}^{(0)}=\theta_k+a_kd_k+b_kd_{k-1},\qquad d_k=\theta_k-\theta_{k-1}.

Unlike fixed extrapolation, RD-PINN fits its coefficients using the parameter history of all completed windows. Each completed window contributes an observation (dj−1,dj,dj+1)(d_{j-1},d_j,d_{j+1}) to the least-squares fit,

(ak,bk)=arg⁡min⁡a,b∑j=3k−1∥adj+bdj−1−dj+1∥22.(a_k,b_k)=\arg\min_{a,b}\sum_{j=3}^{k-1}\Vert ad_j+bd_{j-1}-d_{j+1}\Vert_2^2.

Writing θk−1=θk−2+Δθk−1\theta_{k-1}=\theta_{k-2}+\Delta\theta_{k-1} and θk=θk−2+Δθk\theta_k=\theta_{k-2}+\Delta\theta_k, the same prediction becomes

θk+1(0)=θk−2+(1+ak)Δθk−(ak−bk)Δθk−1.\theta_{k+1}^{(0)}=\theta_{k-2}+(1+a_k)\Delta\theta_k-(a_k-b_k)\Delta\theta_{k-1}.

Ignoring the last term, θk+1(0)=θk−2+(1+ak)Δθk\theta_{k+1}^{(0)}=\theta_{k-2}+(1+a_k)\Delta\theta_k has a form similar to gradient descent, where 1+ak1+a_k is a learning parameter and Δθk\Delta\theta_k controls the direction, and the last term can be considered as a momentum term. In this form, copying the previous window (ak=bk=0a_k=b_k=0) fixes the learning parameter at 1 with no momentum term, while linear extrapolation (ak=1a_k=1, bk=0b_k=0) fixes it at 2 with a momentum coefficient of −1-1. Regression instead learns both coefficients from the completed windows.

On Burgers 1D, regression starts at window 5; the coefficients are nearly stable from window 10 onward, with the mean aka_k at 0.29 through window 40 (Table R1), because the early increments (norms 1.8–2.5 against about 0.4 later) dominate the cumulative least-squares fit. Using only the three most recent windows gives similar accuracy (2.10% versus 2.15%; Table R6 in response to Q3).

Table R1. Fitted coefficients, Burgers 1D

Window kk68102040
aka_k0.350.300.290.290.29
bkb_k−0.05−0.05−0.06−0.05−0.04

Random initialization, linear extrapolation and regression differ in what they take from the completed windows, and the ablation in our response to W4, W5 and Q2 quantifies this difference.

Table R2. Three initializations of window k+1k+1: mean relative L2L_2 error ∥u−uref∥2/∥uref∥2\Vert u-u_{\mathrm{ref}}\Vert_2/\Vert u_{\mathrm{ref}}\Vert_2 (%); epochs and wall-clock as multiples of those of regression

InitializationBurgers 1DBurgers 2DTaylor–GreenEpochsWall-clock
Random: θ∼\theta\sim initializer54.627.129.14.54.5–7.3×7.3\times4.34.3–7.2×7.2\times
Linear extrapolation: θk+dk\theta_k+d_k14.32.775.511.21.2–4.4×4.4\times1.21.2–4.4×4.4\times
Regression (RD-PINN): θk+akdk+bkdk−1\theta_k+a_kd_k+b_kd_{k-1}2.152.463.3311

Random initialization discards the information from completed windows, resulting in errors 9–25 times those of regression and requiring 4.5–7.3 times more training epochs (Table R2). Linear extrapolation uses only the latest parameter increment with a fixed coefficient of 1. It performs similarly to copying on Burgers 2D (2.77% vs. 2.75%) and Taylor–Green (5.51% vs. 5.32%), but substantially worse on Burgers 1D (14.3% vs. 3.28%). In contrast, regression fits both coefficients using parameter increments from all completed windows and updates them as new windows are completed. Unlike fixed linear extrapolation, RD-PINN adapts both coefficients to the observed parameter evolution, achieving the lowest errors across all three problems.

Most time-window PINNs that transfer parameters copy the previous window's weights. Bihlo and Popovych [1] do so and report substantially fewer epochs, and Penwarden et al. [2] explain this benefit. Copying the previous weights makes the initial interface loss zero and brings the PDE residual closer to convergence. None of them uses the parameters of earlier windows. The idea closest to ours outside PINNs is the parameter continuation of Pathak and Paffenroth [3], which extrapolates along a secant of two earlier iterates of an optimization and adjusts the step with a loss-based heuristic. It acts along the iterations of one optimization, not across the windows of a time-dependent solve, and it does not fit its coefficients. To our knowledge, RD-PINN is the first time-window method that fits a predictor of the next window's parameters to the history of all completed windows.

[1] Bihlo, Alex, and Roman O. Popovych. "Physics-informed neural networks for the shallow-water equations on the sphere." Journal of Computational Physics 456 (2022): 111024.

[2] Penwarden, Michael, et al. "A unified scalable framework for causal sweeping strategies for physics-informed neural networks (PINNs) and their temporal decompositions." Journal of Computational Physics 493 (2023): 112464.

[3] Pathak, Harsh Nilesh, and Randy Paffenroth. "Parameter continuation methods for the optimization of deep neural networks." 2019 18th IEEE International Conference on Machine Learning and Applications (ICMLA), 2019: 1637–1643.

Reviewer_mzvDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes REgressive Dual-PINN (RD-PINN) to address two challenges in physics-informed NN: the computational overhead of AD and unreliable predictions ove long time periods. The computational overhead of AD is reduced by using dual numbers, while predictions over long time periods are improved through predicting the initial network parameters for subsequent time windows. Dual-FD is evaluated across 7 PDE problems and ten neural nework architectures. The results show a median speedup of 2.42x.

Critical strengths
  • This paper targets two important challenges in PINNs, and proposes a coherent approach to address them.
  • The experiments cover multiple PDEs and ten PINN architectures. The results are promising in both wall-clock time to reach specified acuracy threshods and in preserving physically meaningful dynamics.
Critical weaknesses
  • Although combining dual numbers and finite-difference approximations show positive results, the aper could better distinguish its methodological novelty from existing hybrid differentiation and parameter extrapolation approaches.
  • The paper compares RD-PINN with sveral temporal-training methods. However, it remains unclear how much improvement arises from histroical parameter regression. A more convincing evaluation would compare regressive initialiation against previous-window weight copying, linear extrapolation and other simple warm-start strategies under identical training configurations.
  • new network parameters are predicted from historical parameter increments. The paper provides limited analysis of when parameter-space regression is reliable. After all, neural-network paramters need not evolve smoothly, particularly for rapidly changing dynamics.
  • The paper reports mixed speedups achieved by Dual-FD on 1D problems. This suggests that the effectiveness of Dual-FD depends on the PDE and numerical settings. More insights into these dependencies could improve the method's practical value.
  • The paper introduces physical index. But it only measures the strength of physical fluctuations, not whether their complete temporal evolution is correct.
  • Some of the recent very relevant methods are not compared against or discussed, such as "DP-PINN+: A dual-phase PINN learning with automated phase division, published in Journal of Computational Science in 2025.
Questions for authors
  • can authors provide a controlled ablation comparing regressive intialization against previous-window weight copying and simple parameter extropolation?
  • How sensitive is regressive initialization to the number and the length of temporal windows?what happens when the parameter changes become irregular?
  • can authors clarify which PDE characteristics and network architectures favour dual-FD over AD?
  • Can authors report temporal errors and additional physical diagnosis to establish wehther RD-PINN captures the correct oscillation frequency and phase?
  • Can authors compare (or at least discuss) the proposed framework against DP-PINN?
9 comments
AuthorsOpenReview ↗

1. Response to W1 (Part 1): Dual-FD versus existing differentiation methods

We sincerely thank the reviewer for the constructive feedback. We appreciate the recognition that the paper targets two important challenges in PINNs with a coherent approach, that the experiments cover multiple PDEs and ten PINN architectures, and that the results are promising both in wall-clock time to a specified accuracy and in preserving physically meaningful dynamics. Below, we carefully address the reviewer's concerns and provide further clarifications and additional evidence to strengthen our work.

1. Response to W1 (Part 1): Dual-FD versus existing differentiation methods

We clarify the distinction between Dual-FD and existing differentiation methods through their computational mechanisms and direct comparisons.

Dual-FD combines two established techniques in a simple way. To our knowledge, no prior PINN method combines first-order dual-number differentiation with a single centered difference of exact first derivatives to compute second-order PDE residuals. Dual-FD avoids nested reverse-mode AD and explicit second-order derivative propagation while keeping first derivatives exact. This simple combination reduces differentiation costs without changing the network architecture or training procedure, as demonstrated across seven PDEs and ten architectures. We regard this simplicity as an advantage, as the operator is easy to adopt in existing PINN code, and the gains come from the combination rather than from additional machinery.

Higher-order forward AD extends first-order forward differentiation using nested dual numbers (forward-over-forward), Taylor-mode AD, or Forward Laplacian propagation. The nested computation of second- and higher-order derivatives slows down the whole process.

Dual-FD takes a different approach: it uses dual numbers to compute only the first-order derivatives and approximates second derivatives through centered finite differences:

uxx(x)≈uxdual(x+h)−uxdual(x−h)2h.u_{xx}(x)\approx\frac{u_x^{\mathrm{dual}}(x+h)-u_x^{\mathrm{dual}}(x-h)}{2h}.

Thus, no second-order derivatives are propagated through the network. For smooth activation networks, direct second-order AD involves σ′′\sigma'' in the PDE residual and requires derivatives up to σ′′′\sigma''' during parameter backpropagation. Dual-FD instead involves only σ\sigma and σ′\sigma' in the residual, requiring derivatives up to σ′′\sigma'' during training. This reduces higher-order differentiation costs.

AuthorsOpenReview ↗

2. Response to W1 (Part 1) (Cont.): Comparison with existing differentiation methods

Forward-mode and higher-order differentiation

SPINN [1] differs from Dual-FD in two respects. First, it applies the dual pass twice (forward over forward) to obtain second derivatives, which places σ′′\sigma'' in the residual, whereas Dual-FD applies it once. Second, it requires a separable architecture with one subnetwork per input coordinate, whereas Dual-FD works on any network. On SPINN's own 3D Helmholtz case, replacing only its differentiation with Dual-FD makes each training step 4.2× faster and reaches the error thresholds 2.9–4.8× sooner, with comparable final errors.

Taylor-mode AD [2] extends forward differentiation to higher orders by propagating Taylor derivative information through network layers. DOF [3] uses Forward Laplacian propagation to compute the gradient and Laplacian directly without constructing the full Hessian. STDE [4] estimates differential operators using randomly sampled Taylor-mode directions, reducing high-dimensional computation at the cost of sampling variance. Cao et al. [5] build an explicit forward-propagation kernel for each PDE residual, which propagates only the derivative states that residual needs. The kernel is specific to the residual and the network, so a new residual needs a new kernel.

Dual-FD requires only PDE-independent first-order forward differentiation rules, so new residuals need no new derivative kernels. On Kovasznay and Taylor–Green, Dual-FD is 1.6–1.8× faster per training step than Taylor-mode AD, 1.9–2.3× faster than DOF, and approximately 3.8–3.9× faster than nested AD. Since STDE is designed for high-dimensional problems, we evaluated both methods on a dd-dimensional Poisson equation. Dual-FD outperforms STDE in per-step runtime up to d=8d=8. For d=3d=3–77, Dual-FD achieves an average speedup of 1.26×, with a maximum of 1.83× at d=3d=3. STDE is faster than Dual-FD at the tested dimensions above d=8d=8. On Burgers 1D, the only case shared with Cao et al., Dual-FD is 1.4× faster than their released code on the same GPU at the same final loss.

Hybrid differentiation. CAN-PINN [6] also combines AD and finite differences. Its upwind approximation combines a FD of network values with an AD-based correction:

ux(x)≈u(x)−u(x−h)h⏟FD+uxAD(x)−uxAD(x−h)2⏟AD correction.u_x(x)\approx \underset{\mathrm{FD}}{\underbrace{\frac{u(x)-u(x-h)}{h}}} + \underset{\mathrm{AD\ correction}}{\underbrace{\frac{u_x^{\mathrm{AD}}(x)-u_x^{\mathrm{AD}}(x-h)}{2}}}.

Here, reverse-mode AD provides a correction to the FD estimate rather than serving as the final first-derivative operator. CAN-PINN computes second derivatives using centered differences of network values. This incorporates neighboring-point information into the residual to improve training with sparse collocation points.

Dual-FD instead obtains exact first derivatives using forward-mode AD and applies FD only to these derivatives, targeting the computational cost of second-order residuals. On Kovasznay and Taylor–Green, Dual-FD is 2.7–4× faster per step, reaches matched accuracy thresholds 1.8–2.3× sooner, and achieves 1.6–10× lower final errors. The methods differ in how they compute both first- and second-order derivatives. CAN-PINN computes first derivatives with a reverse pass over the coordinates, whereas Dual-FD obtains them in one forward dual pass. The second-order difference of network values in CAN-PINN carries a larger error than the single difference of exact first derivatives in Dual-FD.

Dual-FD combines first-order dual-number AD with centered differences of first derivatives. By avoiding nested reverse-mode AD and explicit second-order propagation, it reduces differentiation costs without changing the training procedure, and it requires neither a separable architecture, as SPINN does, nor residual-specific kernels, as Cao et al. do. The same operator applies to all ten architectures in the paper's Table 1.

[1] Cho, Junwoo, et al. "Separable physics-informed neural networks." Advances in Neural Information Processing Systems 36 (2023).

[2] Dangel, Felix, et al. "Collapsing Taylor mode automatic differentiation." Advances in Neural Information Processing Systems 38 (2025).

[3] Li, Ruichen, et al. "DOF: Accelerating high-order differential operators with forward propagation." arXiv preprint arXiv:2402.09730 (2024).

[4] Shi, Zekun, et al. "Stochastic Taylor derivative estimator: Efficient amortization for arbitrary differential operators." Advances in Neural Information Processing Systems 37 (2024).

[5] Cao, Wenbo, Zhe Lu, and Weiwei Zhang. "Verified residual-specific explicit derivative kernels for physics-informed learning and discretized PDE adjoints." arXiv preprint arXiv:2606.29702 (2026).

[6] Chiu, Pao-Hsiung, et al. "CAN-PINN: A fast physics-informed neural network based on coupled-automatic–numerical differentiation method." Computer Methods in Applied Mechanics and Engineering 395 (2022): 114909.

AuthorsOpenReview ↗

3. Response to W1 (Part 2): Regressive initialization as an initialization scheme

Training the network parameters in a new window, we have to initialize them. Previous work initializes the parameters randomly or copies the parameters determined in the previous window. Bihlo and Popovych [1] and Penwarden et al. [2] noticed that the latter approach provides significant speedup. Some can consider that this approach "pre-trains" the network parameters in the previous window. Numerous works have demonstrated how important pre-training is. The proposed regression method is an extension of this direction. When the parameters ak=0a_k=0 and bk=0b_k=0, our regression method becomes the copying approach. In this work, we take the increments of the parameters in the previous windows to adjust the network parameters θk\theta_k and the parameters in all completed windows to determine their coefficients. In other words, same as the copying approach, the proposed method is an initialization scheme. The question is whether this initialization is effective or not. Our experimental results show that, under identical training settings, regression lowers the horizon-mean error relative to copying by 11–37% and the training epochs by 7–13% on Burgers 1D, Burgers 2D and Taylor–Green (our response to W2 and Q1).

To further analyze the regression method, we rewrite the regression formula as

θk+1(0)=θk−2+(1+ak)Δθk−(ak−bk)Δθk−1,\theta_{k+1}^{(0)}=\theta_{k-2}+(1+a_k) \Delta\theta_k-(a_k-b_k) \Delta\theta_{k-1},

where θk−1=θk−2+Δθk−1\theta_{k-1}=\theta_{k-2}+\Delta\theta_{k-1} and θk=θk−2+Δθk\theta_k=\theta_{k-2}+\Delta\theta_k, and Δθk−1\Delta\theta_{k-1} and Δθk\Delta\theta_k are parameter updates through training. Ignoring the term (ak−bk)Δθk−1(a_k-b_k)\Delta\theta_{k-1}, θk+1(0)=θk−2+(1+ak)Δθk\theta_{k+1}^{(0)}=\theta_{k-2}+(1+a_k)\Delta\theta_k has a form similar to gradient descent, where 1+ak1+a_k is a learning parameter and Δθk\Delta\theta_k controls the direction. The last term can be considered as a momentum term. Our approach uses previous windows to determine their weights.

It should be emphasized that we do not assume that the parameters are unique, and the regressed parameters are for initialization only, which require training on the residual loss to obtain the final parameters. Furthermore, we apply a norm constraint to restrict the difference between θk+1(0)\theta_{k+1}^{(0)} and θk\theta_k. Our response to W2 and Q1 isolates the benefit of this initialization against copying and fixed extrapolation under identical training settings.

[1] Bihlo, Alex, and Roman O. Popovych. "Physics-informed neural networks for the shallow-water equations on the sphere." Journal of Computational Physics 456 (2022): 111024.

[2] Penwarden, Michael, et al. "A unified scalable framework for causal sweeping strategies for physics-informed neural networks (PINNs) and their temporal decompositions." Journal of Computational Physics 493 (2023): 112464.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Area Chair and Reviewers,

We thank all reviewers for their time and constructive feedback. In revising the manuscript, we focused on clarifying the contribution of each component and adding the comparisons you requested. Throughout these updates, the Dual-FD operator, the regressive initialization rule (Eqs. 4 and 5), and the main results (Tables 1 and 2) remain unchanged.

The revision makes two main updates.

  • Dual-FD (Sec. 2, Apps. C.5 to C.8): We add comparisons with CAN-PINN, DOF, Taylor-mode AD and STDE.
  • Regressive initialization (Sec. 2, Sec. 3.3, Sec. 4.3, Table 3, Apps. E.3 to E.5): We explain how the regression coefficients are obtained from the parameter changes of completed windows. We also compare regressive initialization with random initialization, copying, and linear extrapolation under same training settings.

We hope these revisions clarify the motivation for RD-PINN and where it is effective, and provide stronger support for our claims. Thank you again for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.