- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes regressive dual-PINN to improve the speed and reliability of physics-informed neural networks for solving PDEs over long time horizons. It introduces two components: Dual-FD, which combines dual numbers for first-order derivatives with finite differences for second-order derivatives, and regressive initialization, which uses historical neural network weights to initialize subsequent time windows. The experiments show promising speedups, reduced memory usage, and improved long-horizon predictions.
Critical strengths
- Dual-FD reduces the computational cost of derivative calculations while maintaining accuracy.
- The method achieves speedups, particularly for two-dimensional PDEs, and reduces GPU memory consumption.
- Regressive initialization preserves important physical behavior over longer time intervals, addressing a known limitation of traditional PINNs.
Critical weaknesses
- Dual numbers, finite differences, and parameter extrapolation are established techniques. The paper does not sufficiently clarify its methodological novelty relative to related approaches.
- The paper lacks direct experimental comparisons with relevant differentiation methods such as DOF, CAN-PINN, and STDE, and temporal approaches such as E-DNN.
- The paper does not clearly isolate the contribution of regressive initialization from Dual-FD. Therefore, it is difficult to determine which component dictates the long-horizon improvements.
- Although numerical errors are discussed, the paper does not deeply investigates how approximation errors and prediction errors accumulate across consecutive time windows.
- The sensitivity of regressive initialization to window size and historical parameter changes is unclear. Additionally, differences in compilation settings between Dual-FD and AD may affect the reported speedups.
Questions for authors
- What is the main methodological novelty of Dual-FD compared with existing hybrid differentiation techniques, and how does regressive initialization differ from conventional parameter extrapolation?
- Can the authors provide comparisons with recent differentiation methods such as DOF and CAN-PINN, as well as temporal approaches such as E-DNN, to better demonstrate the advantages of RD-PINN?
- Can the authors evaluate Dual-FD and regressive initialization independently, including comparisons against simple weight copying and extrapolation, to quantify each component's contribution?
- How do numerical approximation errors and prediction errors propagate across time windows? Does regressive initialization reduce or potentially amplify accumulated errors over longer horizons?
- How sensitive is the method to window size and the quality of historical parameter estimates? Can the authors also report speed comparisons under matched compilation settings?
11 comments
2. Response to W1 (Part 1) (Cont.): Comparison with forward-mode, higher-order and hybrid methods
Forward-mode and higher-order differentiation
SPINN [1] differs from Dual-FD in two respects. First, it applies the dual pass twice (forward over forward) to obtain second derivatives, which places in the residual, whereas Dual-FD applies it once. Second, it requires a separable architecture with one subnetwork per input coordinate, whereas Dual-FD works on any network. On SPINN's own 3D Helmholtz case, replacing only its differentiation with Dual-FD makes each training step 4.2× faster and reaches the error thresholds 2.9–4.8× sooner, with comparable final errors.
Taylor-mode AD [2] extends forward differentiation to higher orders by propagating Taylor derivative information through network layers. DOF [3] uses Forward Laplacian propagation to compute the gradient and Laplacian directly without constructing the full Hessian. STDE [4] estimates differential operators using randomly sampled Taylor-mode directions, reducing high-dimensional computation at the cost of sampling variance. Cao et al. [5] build an explicit forward-propagation kernel for each PDE residual, which propagates only the derivative states that residual needs. The kernel is specific to the residual and the network, so a new residual needs a new kernel.
Dual-FD requires only PDE-independent first-order forward differentiation rules, so new residuals need no new derivative kernels. On Kovasznay and Taylor–Green, Dual-FD is 1.6–1.8× faster per training step than Taylor-mode AD, 1.9–2.3× faster than DOF, and approximately 3.8–3.9× faster than nested AD. Since STDE is designed for high-dimensional problems, we evaluated both methods on a -dimensional Poisson equation. Dual-FD outperforms STDE in per-step runtime up to . For –, Dual-FD achieves an average speedup of 1.26×, with a maximum of 1.83× at . STDE is faster than Dual-FD at the tested dimensions above . On Burgers 1D, the only case shared with Cao et al., Dual-FD is 1.4× faster than their released code on the same GPU at the same final loss.
Hybrid differentiation
CAN-PINN [6] also combines AD and finite differences. Its upwind approximation combines a FD of network values with an AD-based correction:
Here, reverse-mode AD provides a correction to the FD estimate rather than serving as the final first-derivative operator. CAN-PINN computes second derivatives using centered differences of network values. This incorporates neighboring-point information into the residual to improve training with sparse collocation points.
Dual-FD instead obtains exact first derivatives using forward-mode AD and applies FD only to these derivatives, targeting the computational cost of second-order residuals. On Kovasznay and Taylor–Green, Dual-FD is 2.7–4× faster per step, reaches matched accuracy thresholds 1.8–2.3× sooner, and achieves 1.6–10× lower final errors. The methods differ in how they compute both first- and second-order derivatives. CAN-PINN computes first derivatives with a reverse pass over the coordinates, whereas Dual-FD obtains them in one forward dual pass. The second-order difference of network values in CAN-PINN carries a larger error than the single difference of exact first derivatives in Dual-FD.
Dual-FD combines first-order dual-number AD with centered differences of first derivatives. By avoiding nested reverse-mode AD and explicit second-order propagation, it reduces differentiation costs without changing the training procedure, and it requires neither a separable architecture, as SPINN does, nor residual-specific kernels, as Cao et al. do. The same operator applies to all ten architectures in the paper's Table 1.
[1] Cho, Junwoo, et al. "Separable physics-informed neural networks." Advances in Neural Information Processing Systems 36 (2023).
[2] Dangel, Felix, et al. "Collapsing Taylor mode automatic differentiation." Advances in Neural Information Processing Systems 38 (2025).
[3] Li, Ruichen, et al. "DOF: Accelerating high-order differential operators with forward propagation." arXiv preprint arXiv:2402.09730 (2024).
[4] Shi, Zekun, et al. "Stochastic Taylor derivative estimator: Efficient amortization for arbitrary differential operators." Advances in Neural Information Processing Systems 37 (2024).
[5] Cao, Wenbo, Zhe Lu, and Weiwei Zhang. "Verified residual-specific explicit derivative kernels for physics-informed learning and discretized PDE adjoints." arXiv preprint arXiv:2606.29702 (2026).
[6] Chiu, Pao-Hsiung, et al. "CAN-PINN: A fast physics-informed neural network based on coupled-automatic–numerical differentiation method." Computer Methods in Applied Mechanics and Engineering 395 (2022): 114909.
3. Response to W1 (Part 2) and Q1 (Part 2): Regressive initialization versus low-order extrapolation
We would like to clarify what regressive initialization is and how it differs from a low-order extrapolation.
The form. RD-PINN initializes window as
and copying the previous window () and linear extrapolation (, ) are special cases of this form. The distinction lies in how the coefficients are obtained.
How the coefficients are obtained. Unlike conventional extrapolation with fixed coefficients, RD-PINN adaptively fits its coefficients using the parameter history accumulated across completed windows. Every completed window provides one observation consisting of the two preceding parameter increments, and , and the subsequent increment . The coefficients are the least-squares solution over all these observations (Sec. 3.3 and App. A.5 of the paper),
Each completed window adds one observation to the fit, so the fit uses the whole history at negligible cost because of the cumulative update scheme (Eq. 14 and App. A.5 of the paper). The predicted update is bounded in size, with by default, and the predicted parameters are only the starting point of the next window, which is then trained on the PDE residual as before. The two latest increments thus set the directions of the prediction, and the whole history sets how far to move along them.
To see what the regression changes, we rewrite it as
where and are the parameters reached by training. Ignoring the last term, has a form similar to gradient descent, where is a learning parameter and controls the direction, and the last term can be considered as a momentum term. In this form, copying fixes the learning parameter at 1 with no momentum term and linear extrapolation fixes it at 2 with a momentum coefficient of , whereas regression learns both from the completed windows. The regressed parameters are for initialization only and are trained on the residual loss, and a norm constraint restricts the difference between and .
On Burgers 1D, regression starts at window 5 with one observation. Each completed window adds one observation and corrects the coefficients, and from window 9 onward they are nearly stable through window 40 (Table R1), because the early increments, whose norms are 4–6 times larger than the later ones, dominate the cumulative least-squares fit. The coefficients are thus learned from the run as it proceeds, which a rule fixed in advance cannot do.
Table R1. Fitted coefficients window by window, Burgers 1D
| Window | 5 | 6 | 7 | 8 | 9 | 10 | 20 | 30 | 40 |
|---|---|---|---|---|---|---|---|---|---|
| Completed windows used in the fit | 1 | 2 | 3 | 4 | 5 | 6 | 16 | 26 | 36 |
| 0.38 | 0.35 | 0.31 | 0.30 | 0.29 | 0.29 | 0.29 | 0.29 | 0.29 | |
| −0.01 | −0.05 | −0.05 | −0.05 | −0.06 | −0.06 | −0.05 | −0.05 | −0.04 |
1. Response to W1 (Part 1) and Q1 (Part 1): Dual-FD versus existing differentiation methods
We sincerely thank the reviewer for the constructive feedback. We appreciate the recognition that Dual-FD reduces the cost of derivative calculations while maintaining accuracy and lowers GPU memory consumption, with speedups in particular on two-dimensional PDEs, and that regressive initialization preserves important physical behaviour over longer time intervals, addressing a known limitation of traditional PINNs. Below, we carefully address the reviewer's concerns and provide further clarifications and additional evidence to strengthen our work.
We clarify the distinction between Dual-FD and existing differentiation methods through their computational mechanisms and direct comparisons.
Dual-FD combines two established techniques in a simple way. To our knowledge, no prior PINN method combines first-order dual-number differentiation with a single centered difference of exact first derivatives to compute second-order PDE residuals. Dual-FD avoids nested reverse-mode AD and explicit second-order derivative propagation while keeping first derivatives exact. This simple combination reduces differentiation costs without changing the network architecture or training procedure, as demonstrated across seven PDEs and ten architectures. We regard this simplicity as an advantage, as the operator is easy to adopt in existing PINN code, and the gains come from the combination rather than from additional machinery.
Higher-order forward AD extends first-order forward differentiation using nested dual numbers (forward-over-forward), Taylor-mode AD, or Forward Laplacian propagation. The nested computation of second- and higher-order derivatives slows down the whole process.
Dual-FD takes a different approach: it uses dual numbers to compute only the first-order derivatives and approximates second derivatives through centered finite differences:
uxx(x)≈2huxdual(x+h)−uxdual(x−h).
Thus, no second-order derivatives are propagated through the network. For smooth activation networks, direct second-order AD involves σ′′ in the PDE residual and requires derivatives up to σ′′′ during parameter backpropagation. Dual-FD instead involves only σ and σ′ in the residual, requiring derivatives up to σ′′ during training. This reduces higher-order differentiation costs.