The Sigmoid Strikes Back

Physics-informed objectives for second-order equations read a network's second input derivatives. For the model reaction–diffusion equation, we prove that the risk over piecewise-linear networks has infimum zero, yet every minimizing sequence converges in $L^2$ to the source rather than the true solution. Its residual vanishes while its solution error approaches the norm of the solution's Laplacian; for unit-norm high-frequency sources, that error approaches one as the solution norm vanishes. The same order reduction holds at every depth and for general elliptic and fully nonlinear operators, with wrong-limit conclusions under the stated solvability conditions. The regression identity is exact at collocation points, and output-weight gradient descent with fixed piecewise-linear features reaches the spurious limit as their spans become dense. An almost-everywhere Hessian-only penalty is constant on this class. Piecewise-linear energy models also collapse under score matching to a zero score. In contrast, smooth saturating networks approximate sufficiently regular solutions together with their second derivatives. When the residual and boundary misfit control solution error, every minimizing sequence is consistent; logistic sigmoid and hyperbolic tangent networks achieve a quantitative rate.