- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper primarily demonstrates a failure mode caused by inter-point derivatives in fusion-based PINNs, including physics-informed neural operators. Specifically, even when a neural network achieves a small residual error, the resulting solution function may not satisfy the physical laws.
To explain and address this issue, the authors propose treating fusion inputs as fixed context when computing coordinate derivatives, eliminating the corresponding gradient paths through gradient decoupling. They also introduce the Derivative Cheating Intensity (DCI) metric. Extensive experiments across multiple PDEs and fusion architectures demonstrate that an appropriate derivative computation method can prevent models from producing incorrect solutions caused by inter-point interactions.
Critical strengths
The paper identifies a practically meaningful problem. Through computational graph analysis, it reveals a mismatch between residual error and the final derivative fitting in some existing fusion models. Experiments and visualizations demonstrate the practical impact of this problem on solving PDEs.
The correction is simple and easy to implement. Based on a detailed analysis of the computational graph, the authors propose a simple and effective method for removing inter-point effects and provide a reasonable diagnostic metric.
The paper is clearly written and logically organized. Its motivation is clear and well defined, drawing on a reasonable computational graph analysis and experimental results from concrete applications. Based on these results and analyses, the authors develop a corresponding solution and present it through intuitive experimental designs and metrics. Readers can easily understand the core problem and the proposed solution.
Critical weaknesses
The most important weakness concerns where the core significance of the method lies. First, models with inter-point fusion can be divided into two categories: neural operators and PINNs. For neural operators, their capabilities come primarily from training data, namely supervised learning, while physics-informed training serves more as an auxiliary tool. Moreover, because their outputs are functions, their residuals do not necessarily need to be computed through automatic differentiation; finite differences and other methods can also be used. For PINNs, mainstream models currently do not use fusion. Furthermore, some existing fusion models use independently supplied query inputs (x), and therefore do not suffer from the problem described in this paper.
Therefore, I believe the paper needs stronger evidence demonstrating that the problem it addresses is meaningful, rather than merely summarizing a problem that has already been addressed through technical measures in other fusion models or does not actually arise in practice. I think the authors have identified an interesting problem, but there are further questions worth considering and discussing. The paper’s discussion of its central issue currently lacks sufficient depth.
Second, the paper overclaims its theoretical contribution. Regarding the second contribution listed in the introduction, I do not consider it a theoretical contribution if it merely applies the chain rule to derive the computational graph and illustrate a highly intuitive phenomenon.
Questions for authors
Building on the discussion of supervised training in Section 5.1, how does the severity of the identified failure mode change when supervised solution data are incorporated into physics-informed operator learning? Comparisons across data-only, physics-only, and hybrid training would help clarify whether supervision mitigates the problem and how the benefit of decoupling depends on the amount of data and the weight of the physics loss.
Can finite-difference residuals serve as a practical alternative to the proposed correction? Under comparable accuracy and sampling conditions, how do finite differences and gradient-decoupled AD compare in terms of solution accuracy, training time, and memory consumption? How do resolution and discretization errors affect this comparison?
Beyond the comparisons already provided, could the authors compare corrected fusion-based models with pointwise models under matched computational budgets? This would help clarify the practical benefit of fusion after the identified failure mode has been addressed. It would also be valuable to discuss which PDE characteristics or problem settings are most likely to make corrected fusion-based models competitive.
6 comments
[Q1] Regarding the data/physical hybrid-driven issue
Introducing supervised data does not automatically correct the semantic discrepancy in an AD-based PDE residual. In hybrid training, the data loss may compensate for an inaccurate physics constraint, but the underlying residual remains inconsistent. Therefore, a more likely scenario is that introducing incorrect PDE residuals would actually reduce solution accuracy compared with a purely data-driven approach. Importantly, without explicitly identifying Derivative Cheating problem, researchers would typically difficult to realize where goes wrong. This is the significance of our work.
[Q2] Finite differences vs. automatic differentiation
Finite differences can of course serve as an alternative means of evaluating spatial derivatives. However, it will introduce discretization errors and require appropriate choices of stencil, resolution, and boundary treatment, and require collocation points to lie on a regular grid. Therefore, automatic differentiation is the default implementation of PINN follows Raissi et al. (2017), while works that employ finite differences generally explicitly highlight this choice in their experimental settings.
[Q3] Comparison of Fusion-based Model and MLP
We would like to clarify that this require is out of our claim scope. Our paper is a diagnostic study, not a proposal claiming that fusion-based architectures universally outperform point-wise MLPs. Requiring a diagnostic paper to first demonstrate that the target methods it diagnoses outperform other methods is unreasonable. This is like we claiming to have fixed a car, but the reviewer require us to proof that it is faster than other cars. We have never seen a similar precedent in the past diagnostic papers. For example, Krishnapriyan et al. [6] diagnosed gradient pathologies in PINNs without first having to demonstrate that PINNs outperform other numerical methods, while Wang et al. [5] investigated the relationship between network depth and performance in Contrastive RL without first having to demonstrate that Contrastive RL outperforms other RL methods.
We appreciate that the reviewer recognizes the practical relevance of the identified problem and the effectiveness of our correction. However, we believe the current assessment does not adequately account for the evidence already presented in our paper and outside community. We respectfully request that the reviewer reconsider whether the stated weaknesses actually challenge our contribution.
References
[1] W. Zhong et al., Physics-Informed Geometry-Aware Neural Operator, CMAME, 2025.
[2] R. Bischof, 4 Ideas for Physics-Informed Neural Networks that Failed, Towards Data Science (practitioner report), 2023.
[3] S. Ding et al., Physics-informed Neural Operator Learning for Nonlinear Grad-Shafranov Equation, in ICML, 2026.
[4] Issue #9, PINNsFormer repo. https://github.com/AdityaLab/pinnsformer/issues/9.
[5] K. Wang et al., 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities, in NeurIPS, 2025.
[6] A. Krishnapriyan, A. Gholami, Characterizing Possible Failure Modes in Physics-Informed Neural Networks, in NeurIPS, 2021.
Thank you for your response. First, I acknowledge that the identified problem exists. However, considering the practical capabilities of fusion-based models and the fact that many existing fusion models do not exhibit this derivative-cheating issue, I believe its practical significance remains limited. As you emphasize, this is an “academic paper, not a novel,” and its conclusions should therefore be supported by sufficient evidence. In your response, I have not seen additional experimental data supporting the claimed significance of the problem. My questions were raised precisely to clarify this significance, based on my understanding of the paper. In my view, the paper’s main contribution lies in its analysis of the problem rather than in the design of the correction method. Therefore, without sufficient evidence addressing the significance of that problem, I respectfully maintain that my concerns remain unresolved.
We acknowledge the reviewer for your comments. However, we believe that several of your criticisms arise from an incorrect characterization of the scope and contributions of our paper. We would like to clarify the following points.
The reviewer states that we have "identifies a practically meaningful problem" in strengths part, while ask "where the core significance of the method lies" in weaknesses part. We are puzzled by this contradictory assessment.
First, we kindly disagree with your claim that mainstream PINNs use MLPs. Our Sec.3 Related work explicitly identifies a large number of fusion-based PINNs that meet our criteria, including highly cited papers such as PIPN (JCP, cited by 226), PINNsFormer (ICLR'24, cited by 241) and so on. All these works use automatic differentiation to compute PINN residuals, and we have verified that they exhibit Derivative Cheating. More importantly, similar issues have already been sporadic observed by community, for example,
However, none of the above works correctly location the cause of this problem, while our contribution exactly lies in the identification, formalization, and causal validation of this mechanism. Therefore, the reviewer's claim that this problem "does not actually arise in practice" is clearly invalid.
Furthermore, neural operators and PINNs are not mutually exclusive categories. Even among neural operators, there are numerous examples of using automatic differentiation to compute residuals. Even if some methods primarily rely on data-driven learning, as long as they incorporate AD-based PDE residuals, correcting semantic errors in those residuals is necessary. Therefore, the failure mode we identify can be expected to affect a substantial proportion of work in the PINN community. More importantly, our work addresses a fundamental issue of correctness rather than incremental performance gains. Therefore, we believe that compared with other PINN works that propose yet another architecture or optimizer, the significance of our work is clearly greater.
We want to argue the reviewer's assessment that our "discussion of its central issue currently lacks sufficient depth." We have already provided a complete chain of evidence regarding the manifestations, causes, and solutions of this failure mode, while each additional topic you request could independently constitute another ICLR paper. These could be interesting extensions of our work, but addressing such extensive derivative topics within a single paper is impractical and would not help strengthen our core claim.
We respectfully direct the reviewer to the ICLR 2027 Reviewer Guidelines (https://iclr.cc/Conferences/2027/ReviewerGuidelines), which define contribution in terms of whether a submission brings new knowledge to the community, rather than whether it applies sophisticated mathematical theories. We have never claimed to have reinvented the chain rule. Our contribution lies in identifying this failure mode for the first time, while the chain rule is merely a theoretical tool employed in our analysis. We argue that it is unreasonable for the reviewer to question our contribution simply because we used a chain rule decomposition.
For example, the NeurIPS 2025 Best Paper [5] did not propose any new algorithm, but merely increased the number of network layers in RL to 1024, which is an operation that everyone in machine learning knows how to perform. Nevertheless, they challenged the conventional design practice of using shallow neural networks in traditional RL. Therefore, the conference committee and reviewers still value the significant contribution of this paper.