acceptodds
Under review as a conference paper at ICLR 2027

The Sigmoid Strikes Back

Abstract

Physics-informed objectives for second-order equations read a network's second input derivatives. For the model reaction–diffusion equation, we prove that the risk over piecewise-linear networks has infimum zero, yet every minimizing sequence converges in to the source rather than the true solution. Its residual vanishes while its solution error approaches the norm of the solution's Laplacian; for unit-norm high-frequency sources, that error approaches one as the solution norm vanishes. The same order reduction holds at every depth and for general elliptic and fully nonlinear operators, with wrong-limit conclusions under the stated solvability conditions. The regression identity is exact at collocation points, and output-weight gradient descent with fixed piecewise-linear features reaches the spurious limit as their spans become dense. An almost-everywhere Hessian-only penalty is constant on this class. Piecewise-linear energy models also collapse under score matching to a zero score. In contrast, smooth saturating networks approximate sufficiently regular solutions together with their second derivatives. When the residual and boundary misfit control solution error, every minimizing sequence is consistent; logistic sigmoid and hyperbolic tangent networks achieve a quantitative rate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
physics-informed neural networks, activation functions, piecewise-linear networks, rectified linear unit, sigmoid, tanh, consistency, collocation, score matching, approximation rates

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TAYdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper produces a proof that second order linear PINNs trained with ReLU (or similar activation functions) do not converge to the true solution even if their residual vanishes. They show that the solutions converge to a "wrong limit" and they extend these results to some other equation classes.

Critical strengths

None.

Critical weaknesses
  • The results of this paper are trivial and constitute what I consider easy exercises for a first-year graduate PDE class. They do not rise to the level of significance required by a paper.
  • For example: Lemma 3.1 is correct but follows immediately from elementary results. Differentiate a ReLU neural net and note that the second derivative vanishes A.E.. Then trivially if uu is this neural network then Lu=−div(A∇u)+b⋅∇u+cu=(b−∇⋅A)⋅∇u+cuLu = -\mathrm{div}(A \nabla u)+b\cdot \nabla u + cu = (b - \nabla \cdot A)\cdot \nabla u + cu. Theorem 3.2 consists of textbook arguments (from Evans) and these results follow from the observation in Lemma 3.1 combined with routine techniques like H1H^1 density, the trace theorem, and the triangle inequality. Lemma 3.1 and 3.2 already capture the general difficulty of the results in this paper.
  • The results of the paper have limited relevance to practical PINN training since ReLU is not a standard activation function in the PINN literature (the original PINN paper by Raissi et al. (2019) uses tanh, which is the adopted standard).
Questions for authors
  1. The first line of the introduction says "The default activation in contemporary deep learning is piecewise linear". Can you substantiate this claim? PINNs don't typically use ReLU as the activation function (I've always assumed because they have vanishing Hessian), and the broad deep learning community no longer uses ReLU. In 2026 I would argue that ReLU is an antiquated activation function.
  2. What do the authors consider technically novel about their work? I.e. observations which go beyond using \nabla^2 u = 0 and standard tools from an introductory graduate PDE course? What are the practical applications of this work?
  3. The proof of Lemma 3.1 is incorrectly labeled as a Theorem in the appendix.
  4. Why is this classified as "foundation or frontier models, including LLMs".
Reviewer_uuBaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper makes the observation that the ReLU activation function cannot be used if the loss function involves derivatives of the network with respect to its input. This is applied to argue that smooth activation functions must be used for PINNs and score matching.

Critical strengths

The paper is correct. Its claims and theorems are clearly true.

Critical weaknesses

The main observation made in the paper, namely that the ReLU has second order derivatives which vanish identically almost everywhere, is trivial. This has clearly been known for a long time and the community studying PINNs has used smooth activation functions for exactly this reason. Other modifications of the ReLU, such as the ReLU$^k$, have also been introduced to overcome this difficulty in the solution of PDEs.

The presentation on score matching is wrong, or at least a very non-standard variant of score-matching is discussed. In particular, typically the neural network is used to parameterize the score itself (i.e. the gradient directly) and not the energy function.

There are no supporting experiments showing the better performance of the sigmoid that is predicted.

Questions for authors

Can you give a reference which uses the variant of score matching that you present?

Reviewer_zb4cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on a potential problem with the commonly used ReLU activation function in physics-informed neural networks. The authors show that ReLU networks can converge to an incorrect solution even when the training loss approaches zero. Specifically, the paper considers neural networks trained to solve partial differential equations (PDEs). Since ReLU is piecewise linear and its second derivative is zero almost everywhere, the original PDE objective can effectively be reduced to a different learning objective, causing the network to learn an incorrect function. In contrast, smooth activation functions such as Sigmoid and Tanh do not suffer from this specific limitation and can approximate the correct solution under suitable conditions. The authors theoretically prove that ReLU networks can converge to an incorrect limit and further extend this analysis to more general PDE operators and score-matching objectives. They also demonstrate that increasing network depth cannot resolve this issue for piecewise-linear activations, since their second derivatives remain zero almost everywhere regardless of depth. Finally, the authors provide a theoretical analysis showing that networks with smooth activation functions such as Sigmoid and Tanh can consistently approximate the correct solutions under appropriate assumptions.

Critical strengths

1: The paper provides an interesting perspective on the limitations of ReLU, which is widely used in modern neural networks.

2: The authors provide a comprehensive theoretical analysis explaining why ReLU fails in certain PDE-solving tasks and how smooth activation functions can overcome this limitation under suitable conditions.

Critical weaknesses

1: The authors do not clearly distinguish their contributions from previous studies. Some of the theoretical findings have already been discussed in prior work, and the novelty of the proposed analysis needs further clarification.

2: The paper lacks numerical experiments to validate the theoretical findings, making the results less convincing from a practical perspective.

3: The theoretical analysis focuses on specific training settings, which limits its generalizability to more complex neural networks and training procedures.

Questions for authors

1:Can the authors further clarify the main differences between this work and previous studies in the related work section? What are the key novel contributions compared with existing approaches?

2:Can the authors provide numerical experiments to further validate the theoretical findings? The current paper relies entirely on theoretical analysis, and additional empirical evidence would help demonstrate the practical relevance of the proposed conclusions.

1 comment

I really appreciate the detailed response. However, I think the paper still need further improvement. I hope the author can revisit all the reviews and improve their manuscirpt

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.