acceptodds
Withdrawn from ICLR 2027

Restoring Cross-Step Feedback in Physics-Informed Neural Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Physics-Informed Neural Networks (PINNs) are mesh-free solvers for partial differential equations (PDEs), yet their training often suffers from two recurring difficulties. One is loss imbalance, where the gradients of the PDE residual and the boundary or initial losses can differ by orders of magnitude, complicating loss weighting. The other is poor internal propagation, where the backbone's layer-to-layer updates can make sharp or high-frequency solution features difficult to learn. Existing remedies largely address these issues separately, but adaptive weighting can remain sensitive to short-term fluctuations, while stronger backbones can still underfit difficult solution structures. These challenges motivate investigating decayed signal accumulation along both network depth and training time. We propose CSF-PINN (Cross-Step Feedback Physics-Informed Neural Network), which applies a common leaky-integral design at the architecture and loss-weighting levels. The Leaky-Integral Residual Block (LIRB) aggregates transformed features across depth through a recurrence equivalent to a damped hidden-state update with an initial-representation skip connection, while Integral-Controlled Weighting (ICW) combines instantaneous gradient statistics with a ratio of leaky integrals of square-root losses. Across five benchmark PDEs, CSF-PINN achieves the lowest error among all compared methods, including a 67.6% reduction in relative error over the strongest baseline on Wave.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Physics-Informed Neural Networks, Partial Differential Equations, Deep Learning, Scientific Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GWYqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CSF-PINN to address loss imbalance (P1) and poor feature propagation (P2). It combines LIRB, a damped initial-skip residual architecture, with ICW, a loss-weighting rule based on gradient statistics and accumulated square-root losses. The authors connect both through a common leaky-integral recurrence. They provide recurrence and local-response analyses and evaluate five PDE benchmarks with component ablations and auxiliary studies.

Critical strengths

None.

Critical weaknesses

1. Limited novelty and no demonstrated joint-design principle

The related-work positioning is inadequate. Wang et al. [1] already combine a new PINN architecture with gradient-statistics-based learning rate annealing (LRA), including a four-way ablation. Francis et al. [2] combine adaptive residual connections, scheduled loss weighting, and SOAP. LIRB reduces exactly to zk+1=αzk+(1−α)z0+αTk(zk)z_{k+1}=\alpha z_k+(1-\alpha)z_0+\alpha T_k(z_k). Initial-representation skips and learnable residual scaling have clear precedents [2, 3], although the complete architectures differ. ICW reuses the gradient-statistics ratio of [1] and combines it with loss-history averaging, an established direction exemplified by ReLoBRaLo [4]. Its specific hybrid rule remains heuristic. Under a consistently rescaled condition loss Lc↦sLcL_c\mapsto sL_c, its two branches scale as s−1s^{-1} and s1/2s^{1/2}, respectively, without a clear justification.

Crucially, the proposed "unification" amounts to applying similar exponential recurrences along depth and training time. The paper identifies no substantive coupling between LIRB and ICW. Its theoretical results do not establish improved high-frequency learning or stability of their joint optimization. Thus, the claimed unified contribution is not supported beyond combining familiar mechanisms.


2. Inadequate comparisons and no evidence of a distinctive combination benefit

The main evaluation compares weighting methods on an MLP against LIRB+ICW, changing both factors simultaneously. Tables 2 and 10 examine only selected PINNsFormer/PirateNet combinations with SA-PINN or DBPINN. Missing controls include LIRB with LRA, ReLoBRaLo, or ConFIG, and alternative architectures with ICW. Published configurations such as PirateNet+LRA+SOAP [5] and PINNsFormer+DCGD [6] are also omitted. A matched-budget architecture-by-training-method comparison is necessary to establish competitive superiority.

The component ablations show improvement over either component alone, but do not establish a distinctive interaction. Let E00E_{00}, E10E_{10}, E01E_{01}, and E11E_{11} denote the errors of the baseline, LIRB only, ICW only, and LIRB+ICW, respectively. Under a multiplicative reference for independent improvements, define

R=E00E11E10E01.R=\frac{E_{00}E_{11}}{E_{10}E_{01}}.

Here, R<1R<1 indicates a greater improvement than this reference, while R>1R>1 indicates a smaller improvement. Four of five benchmarks have R>1R>1. Although this is only a descriptive diagnostic, the results do not substantiate the claimed synergy.

The numerical ICW hyperparameters, repetition details, and main training schedules are also insufficiently documented. Given the limited methodological novelty, these experimental shortcomings substantially weaken the paper's contribution.

Questions for authors
  1. Clarification: What distinguishes the claimed joint-design principle from prior architecture-plus-weighting approaches [1, 2]? What links LIRB and ICW beyond their similar recurrences?

  2. Evidence needed: Why omit established configurations [5, 6]? Does the improvement persist with matched-budget crossed comparisons, including LIRB+LRA/ReLoBRaLo/ConFIG and PirateNet+ICW?

  3. Clarification: What exact ICW hyperparameters, seeds, run counts, and optimizer schedules were used? Is there paired statistical evidence for an interaction between the components?


References

[1] Wang et al., "Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks". SIAM Journal on Scientific Computing, 2021.

[2] Francis et al., "An Adaptive Residual-Structured and Scheduled Loss Weighting Framework for Enhancing Physics-Informed Neural Networks in Solving Partial Differential Equations". Communications in Nonlinear Science and Numerical Simulation, 2026.

[3] Chen et al., "Simple and Deep Graph Convolutional Networks". ICML, 2020.

[4] Bischof et al., "Multi-Objective Loss Balancing for Physics-Informed Deep Learning". Computer Methods in Applied Mechanics and Engineering, 2025.

[5] Wang et al., "Gradient Alignment in Physics-Informed Neural Networks: A Second-Order Optimization Perspective". NeurIPS, 2025.

[6] Hwang et al., "Dual Cone Gradient Descent for Training Physics-Informed Neural Networks". NeurIPS, 2024.

Reviewer_hoZnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tries to solve two critical limitations of physics informed neural networks. The two proposed techniques tackles each challenge separately. They provide substantial improvements. This paper is rather empirical.

Critical strengths
  • Big improvements on many 1D or 2D PDEs.
  • Some mathematical qualifications for their method, mostly for correctness rather than theoretical advantages.
  • Clear ablation studies to show the effectiveness of the proposed methods.
Critical weaknesses
  • The lack of theoretical analysis of where the advantage comes from.
  • There are many hyperparameters used in this work, such as alpha and beta in ICW. It's unclear how to set them and whether they are easy to tune (sensitive or not).
  • A broader evaluation would be appreciated, e.g., Helmholtz with high wave numbers, Navier-Stokes.
  • The authors blame current approaches for addressing the two challenges separately. But in fact the authors also use two orthogonal techniques. It's not clear whether each technique is the best in its category.
Questions for authors

Apart from the weaknesses:

  • How is z0 initialized? In some cases, the authors specified that they use x; in some cases, it is omitted. Getting z may involve up-projection to a higher dimension and I wonder how it is done.
  • How do ICW's internal states, e.g., P_c and lambda_c, change during the PINN training? Could there be some insights/trends?
Reviewer_GicGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors introduce a framework that integrates a leaky-integral design into both the network architecture and the loss-weighting mechanism. They mostly address two challenges in PINNs: loss imbalance and poor internal signal propagation. Their approach is to balance loss terms by combining gradient statistics with a history of accumulated losses. The method is validated on five simple benchmark PDEs, outperforming a wide range of PINN network baselines; but not compared against random feature methods or classical approaches.

Critical strengths
  • The method achieves the lowest error across multiple diverse PDE benchmarks, with significant gains on the wave equation; this comparison is performed against neural network methods, not the state of the art (classical methods).
  • The conceptual link between architectural propagation and loss weighting via a shared leaky-integral accumulation rule is an intuitive and novel contribution.
  • The authors provide thorough ablation studies that isolate the contributions of their additions.
  • Limited theoretical analysis or results; the analysis is only in the appendix but at least exists.
Critical weaknesses
  • The evaluation focuses on fixed-PDE training in simple cases; the applicability to parameterized PDEs or in higher dimension is not demonstrated.
  • No comparison to the state of the art for the given PDEs, which are all easily solvable by classical methods (much faster and more accurately). This of course is a common theme in PINN research, but should be done regardless; otherwise a reader not familiar with the PDEs / methods may think that the method is the best for these PDEs. If not classical methods, at least a comparison to the best network models trained with random features for these simple PDEs could have been presented (several exist:
  • https://doi.org/10.1016/j.cnsns.2025.109343
  • https://proceedings.iclr.cc/paper_files/paper/2026/hash/ab435f7130455143fd9489f389777552-Abstract-Conference.html ).
Questions for authors
  • How does the computational overhead of maintaining the auxiliary integral states compare to standard PINNs in terms of training time?
  • Would the method be applicable to higher-dimensional PDEs, and are there expectations that the "decay" mechanism needs to be adjusted for increased dimensionality?
  • How fast and accurate can the PDEs solved when using random features or classical methods; and what benefit do the authors see in comparison to those approaches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.