The RKHS Cost of Neural Activations for Sparse Variational Gaussian Processes

Mean functions of sparse variational Gaussian processes are equivalent to feed-forward neural networks, with neurons determined by the covariance between the interdomain inducing variables and the GP function values. We investigate whether inducing variables can be chosen so that these cross-covariances are neurons with typical activation functions, such as ReLU. This requires the desired neurons to lie in the prior's reproducing kernel Hilbert space (RKHS). We study their membership and variational cost. Under the stated growth and parameter assumptions, in dimension $d\ge 2$, a neuron with non-zero weight and bias in the parameter support belongs to its own single-layer NNGP RKHS if and only if the activation is polynomial. Under Gaussian input and parameter measures and the stated regularity assumptions, an activation whose first non-zero derivative jump has order $s$ yields neurons outside any fixed trace-class candidate RKHS for almost every parameter once $d\ge 2s+1$. Smooth non-polynomial neurons can belong to suitable RKHSs, but their scale-invariant RKHS cost grows superpolynomially in probability under the stated fan-in and bias laws. This cost obstruction also holds for random neural-network means. In Gaussian regression, such means are either inadmissible or, under a polynomial combined prior-variance and data budget, have a lower ELBO than their centred counterparts with high probability. These results identify when exact neural-network mean representations are feasible in sparse GPs and suggest that activation-shaped inducing variables may be most promising in low-dimensional settings.