acceptodds
Under review as a conference paper at ICLR 2027

The RKHS Cost of Neural Activations for Sparse Variational Gaussian Processes

Abstract

Mean functions of sparse variational Gaussian processes are equivalent to feed-forward neural networks, with neurons determined by the covariance between the interdomain inducing variables and the GP function values. We investigate whether inducing variables can be chosen so that these cross-covariances are neurons with typical activation functions, such as ReLU. This requires the desired neurons to lie in the prior's reproducing kernel Hilbert space (RKHS). We study their membership and variational cost. Under the stated growth and parameter assumptions, in dimension , a neuron with non-zero weight and bias in the parameter support belongs to its own single-layer NNGP RKHS if and only if the activation is polynomial. Under Gaussian input and parameter measures and the stated regularity assumptions, an activation whose first non-zero derivative jump has order yields neurons outside any fixed trace-class candidate RKHS for almost every parameter once . Smooth non-polynomial neurons can belong to suitable RKHSs, but their scale-invariant RKHS cost grows superpolynomially in probability under the stated fan-in and bias laws. This cost obstruction also holds for random neural-network means. In Gaussian regression, such means are either inadmissible or, under a polynomial combined prior-variance and data budget, have a lower ELBO than their centred counterparts with high probability. These results identify when exact neural-network mean representations are feasible in sparse GPs and suggest that activation-shaped inducing variables may be most promising in low-dimensional settings.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Gaussian processes, sparse variational Gaussian processes, reproducing kernel Hilbert spaces, variational inference

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_AxN7OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether standard neural-network neurons can be used exactly as inducing features in sparse variational GPs. This requires the neurons to belong to the GP prior’s RKHS.

Under the stated assumptions, a neuron belongs to its own single-layer NNGP RKHS only when its activation is polynomial. The paper then considers other kernels. It proves non-membership results for activations with derivative jumps and shows that smooth non-polynomial neurons have very large RKHS costs as dimension increases. It extends the cost result to random networks and derives a consequence for the Gaussian-regression ELBO.

Critical strengths
  1. The paper considers general candidate kernels.

    Theorem 4 is more interesting to me than a non-membership result for one particular neural kernel. Under its assumptions, changing to another fixed trace-class covariance does not solve the problem for a positive-measure set of neurons.

  2. The RKHS cost is connected to the variational objective.

    The ELBO result gives a concrete reason to care about the norm. Under the stated budget, the gain in data fit cannot pay for the cost of exactly reproducing a typical random network mean. The network result also checks that cancellation between neurons does not invalidate this argument.

Critical weaknesses

I did not find a critical flaw in the mathematical results. I have two smaller concerns:

  1. The size of the cost at finite dimensions is unclear. Theorem 5 gives an asymptotic result, but I had trouble judging how large the cost would actually be for a standard activation. Equation (94) provides a bound, though it still depends on the distribution of the Hermite coefficients. A worked example for tanh or Softplus would help show when the bound becomes informative and how much it tells us at realistic dimensions.

  2. FMGP [1] is missing from the related work. As far as I understood FMGPs, it also uses the RKHS formulation of SVGPs to work with prescribed neural-network means, so it seems directly relevant. Its approximation argument on compact sets addresses a different question from exact RKHS membership. Discussing that difference would help explain how the results here relate to an existing construction, especially since FMGP keeps the mean fixed while fitting uncertainty.

Neither point gives me a strong reason to reject the paper, but both would make its practical relevance easier to assess.

[1] Luis A. Ortega, Simón Rodríguez-Santana, and Daniel Hernández-Lobato. Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learning. arXiv:2412.04177, 2024.

Questions for authors
  1. Could you work out the bound in Equation (94) for tanh or Softplus? At roughly what dimensions does it become informative? This would help me judge how large the representation cost is in practice.

  2. Could you discuss the connection to FMGP? Both papers use the RKHS formulation of SVGPs, but FMGP uses approximation on compact sets and keeps the mean fixed while fitting uncertainty. Which conclusions here apply to that setting?

3 comments

Thank you very much for your review! We greatly appreciate your recognition of the generality of the theoretical results and the connection to variational inference. You raised two points, namely how large the cost is at finite dimensions and the missing comparison with FMGP. We hope to have addressed both in the revision. The new experiments are summarised in the general response, so here we focus on your specific questions.

Working out the bound for tanh⁡\tanh and Softplus (Question 1)

The bound you refer to as Equation (94) is Equation (92) in the revision. We have made it an explicit result. The new Proposition 16 (Appendix H.4) states that, at any fixed dimension dd and for every fixed kernel TT, Pr⁡{CT≤B}≤Fd,n(η)+Bη ad,nmd,n,Fd,n(η)=Pr⁡{cn(Rd,b)2≤η}.\Pr\{C_T\le B\}\le F_{d,n}(\eta)+\frac{B}{\eta\,a_{d,n}m_{d,n}},\qquad F_{d,n}(\eta)=\Pr\{c_n(R_d,b)^2\le\eta\}. As you noted, it depends on the distribution of the Hermite coefficients cnc_n. We estimate these distributions numerically, then choose the degree n≤48n\le48 and threshold η\eta that give the strongest bound. We use 16,384 parameter draws with a 99% confidence allowance that holds across all reported cases. Inputs are N(0,Id)\mathcal N(0,I_d), weights N(0,Id/d)\mathcal N(0,I_d/d) and biases N(0,1)\mathcal N(0,1).

Result (Table 3). The table gives a cost that at least 90% of neurons exceed, under every fixed kernel.

ddtanh⁡\tanhSoftplus
80.00620.0073
160.0160.019
322350.10
643.5×10133.5\times10^{13}2.4×1032.4\times10^{3}

When is it informative? The table should be compared with an elementary bound. Every eigenvalue of a fixed kernel is at most its trace, so every neuron satisfies CT(θ)≥∥σθ∥L22C_T(\theta)\ge\|\sigma_\theta\|^2_{L^2}. The 10th percentile of this is about 0.36 for tanh⁡\tanh and 0.18 for Softplus. Proposition 16 therefore tells us nothing new up to d=16d=16, becomes informative and substantial for tanh⁡\tanh at d=32d=32, and becomes informative for Softplus at d=64d=64, where both bounds are large.

Appendix H.4 now states this comparison explicitly. We do not claim these are exact crossover dimensions, since they depend on the weight and bias distributions.

A sharper picture for a specific kernel. A bound that holds for every kernel is necessarily conservative. Under the square-root NNGP prior, which has the lowest average cost of any fixed kernel, median costs are already large at moderate dimensions. At d=32d=32, the median cost is about 5,100 for Softplus and 3.7 billion for tanh⁡\tanh (Table 2, Figure 1). Section 6 also shows the effect on the ELBO, as summarised in the general response.

Relation to FMGP (Question 2)

Thank you for pointing us to this paper, which is indeed very relevant! We now cite this work in Section 2.4. We see the two papers as complementary.

  • FMGP shows that, with a universal kernel, sparse GP means are flexible enough to approximate a pretrained network to any accuracy on a compact set. It uses this to equip the network with predictive uncertainty while keeping its predictions.
  • Our paper studies a different quantity, namely the RKHS norm of a mean, which is what it costs in the ELBO when the GP has to represent and learn that mean itself.

Our conclusions relate to the FMGP setting as follows.

  • Proposition 6 holds for any SVGP posterior, so any RKHS representation of a mean contributes 12∥h∥H2\frac12\|h\|^2_{\mathcal H} to the KL term. FMGP takes the network's predictions as its mean and learns only the uncertainty around it, so this cost is not something FMGP needs to pay. Our results therefore do not affect its training.
  • The divergence of norms for increasingly accurate approximations, noted in Section 2.4, is a general property of RKHS approximation. It matters when a GP must learn a mean under the ELBO, a situation FMGP avoids by design.
  • Our high-dimensional results are stated on Rd\mathbb R^d with Gaussian inputs, and concern randomly drawn networks. They do not describe a specific pretrained network on a compact domain, and they do not bear on FMGP's goal of fitting uncertainty around a fixed mean.

In this sense, FMGP fits naturally with our conclusions. Rather than paying for an expensive mean inside the GP, it takes the network's mean as given and focuses on uncertainty.

Thank you again for your helpful suggestions!

Thank you for the detailed response and the additional experiments. The finite-dimensional results and the discussion of FMGP address my questions, and I maintain my positive assessment.

Thank you for your engagement throughout the discussion period! We are glad our revisions successfully address your questions, and we appreciate your positive assessment.

Reviewer_gDqQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the theoretical capacity of sparse Gaussian processes to precisely replicate standard neural network neurons, revealing fundamental limitations in achieving exact representation. Specifically, the study finds that ReLU-like neurons generally fail to satisfy the mathematical conditions required to belong to the necessary Reproducing Kernel Hilbert Space (RKHS). Conversely, while smooth non-polynomial neurons are mathematically representable within this space, they introduce representation costs that scale super-polynomially with dimensionality.

Critical strengths
  1. The theoretical analysis in this paper is solid. For example, the proof of Theorem 1 is convincing. The Fourier argument is convincing. A nonpolynomial ridge has nonzero frequency support on its weight line. The crucial step is the uniform estimate in equation (23): restricting an angular cutoff to a fixed frequency line avoids exploding transverse derivatives. This supports dominated convergence. The polynomial direction correctly uses finite-dimensionality and bounded evaluation.

  2. The theory work proposes an interesting connection to variational inference: It translates RKHS representation costs into ELBO penalties and controls possible cancellation in random network means.

Critical weaknesses

The main drawback is the lack of empirical analysis to support the theoretical results.

More specifically,

  1. The paper does not quantify the cost of approximate neuron representations, which may remain inexpensive despite the exact-representation obstruction.

  2. The asymptotic results provide no explicit crossover dimensions or useful activation-specific cost estimates. A simple synthetic experiment may help.

  3. The strongest cost results assume fixed priors and randomly sampled networks. Their relevance to trained networks and jointly learned kernels remains unclear. Similarly, a simple toy datasets/example may help readers better understand the practical implications of the theory.

Questions for authors
  1. Can authors provide simple empirical example for some of the conclusions?

  2. Which conclusion applies directly to a trained DGP, rather than a prescribed random shallow-network mean?

1 comment

Thank you for your review, and for your positive assessment of the theory! Your main concern was the lack of empirical support, that is, how large the costs are in practice, whether approximation avoids them, and which results apply to a trained deep GP. We address each in turn.

"Can authors provide simple empirical example for some of the conclusions?"

Yes. The new Section 6 and Appendix H, summarised in the general response, show the following.

  • How large the costs are. Even under the kernel with the lowest average cost, median costs at d=32d=32 are about 27,000 for sigmoid, 5,100 for Softplus and 3.7 billion for tanh⁡\tanh (Figure 1, left).
  • When the costs become unavoidable. For every fixed kernel at d=64d=64, at least 90% of sigmoid, tanh⁡\tanh and Softplus neurons each cost more than 2,000 (Proposition 16, Table 3). In lower dimensions this general bound is weak, so we do not claim a universal threshold.
  • A simple toy example. We generate regression data from a single known neuron and let the ELBO choose only its output weight, whose true value is 1. For Softplus with 512 data points, the typical best weight falls from close to 1 at d=16d=16 to below 10−510^{-5} at d=64d=64 (Figure 1, right). So even the correct neuron gets switched off once it becomes too expensive. Appendix H.7 repeats this with the prior and noise variances fitted by exact GP inference, using enough data that the exact GP still beats a pure-noise model. The learned amplitude is still suppressed, falling to 0.07 for cosine and 0.19 for Softplus at d=40d=40 (Figure 7).

Approximate representations

We agree that this matters. Our obstructions concern exact non-polynomial neurons, so approximation can certainly be much cheaper, though in our experiments the largest savings needed a kernel designed for it. At d=32d=32, allowing 10% error under the original kernel still leaves median costs of at least 478 for sigmoid, 539 for Softplus and a million for tanh⁡\tanh (Table 4). Kernels designed for low-degree polynomial approximations reduce average costs to about 2.5, 50 and 670, at errors below 10% (Appendix H.5), though such a kernel covers only a finite-dimensional space. These two constructions are illustrations, not an exhaustive study. There are many ways to approximate, possibly with other kernels, and we see exploring them as an important direction for future work.

"Which conclusion applies directly to a trained DGP?"

A trained DGP differs from the setting of our probabilistic results in three ways. Its parameters come from optimisation rather than a random draw, its kernel may be learned, and it chooses its own mean rather than being asked to reproduce a given network. We have added a paragraph to Section 5, "Implications for trained networks and adapted kernels", to make the following explicit.

Results that apply directly

  • The ELBO cost (Proposition 6). In any fitted SVGP, with any kernel, learned or not, the mean pays exactly 12∥h∥H2\frac12\|h\|^2_{\mathcal H} in the KL term. In a DGP this holds for every output of every layer.
  • Theorem 1. If a layer uses the NNGP kernel of its own activation, no trained neuron with a non-polynomial activation and non-zero weight can be an inducing cross-covariance.
  • Theorem 4. For a kernel fixed before training, trained ReLU-type neurons are excluded almost surely once the input dimension is large enough (d≥3d\ge3 for ReLU), provided their parameters have a density.

Results that do not apply directly

  • Theorem 5, Proposition 7 and Theorem 8. These describe neurons and networks with randomly drawn parameters, under a kernel not adapted to them. Trained parameters need not follow such a distribution, and a kernel learned together with the network could adapt to it. We expect similar costs for trained networks whose features are not specially aligned with the kernel, but proving this would need further assumptions.

In practice, our results do not say that trained deep GPs fail. An optimiser that is free to choose its mean will simply avoid expensive neurons. What they say is that an SVGP cannot cheaply reproduce a typical high-dimensional network exactly, so the ELBO pushes a trained model towards cheaper means instead, as the toy example above illustrates. Learning the kernel's amplitude together with the noise variance does not prevent this (Appendix H.7).

We hope these additions address your concerns! Thank you again for your review.

Reviewer_o3ZpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when neural activations can serve as inducing-variable cross-covariances in sparse variational Gaussian processes (SVGPs), connecting their admissibility to reproducing kernel Hilbert space (RKHS) membership and their variational cost to RKHS norms. Beyond characterizing membership in the activation’s own neural network Gaussian process RKHS, it establishes broader limitations for arbitrary fixed trace-class candidate priors. Under explicit assumptions, activations with finite derivative jumps exhibit almost-everywhere nonmembership above a dimension threshold, while smooth non-polynomial activations incur super-polynomial growth in scale-invariant RKHS cost. The paper extends the cost obstruction to random finite-width network means and derives consequences for their evidence lower bound (ELBO) under a polynomial combined prior-variance and data budget.

Critical strengths

1. Coherent analytical framework.

The paper organizes its analysis around a natural progression from RKHS membership to representation cost. From the SVGP perspective, it first asks whether neurons with standard activations can serve as valid inducing cross-covariances, then examines their RKHS cost and its implications for the variational objective. This distinction between representability and variational affordability provides a coherent conceptual framework. The main results appear mathematically well supported, with explicit assumptions that clarify their scope.

2. Meaningful theoretical generalization.

The paper extends existing kernel-specific analyses to broader classes of candidate priors. Theorem 4 considers any fixed trace-class candidate covariance, without requiring rotational invariance, and establishes almost-everywhere nonmembership above an activation-dependent dimension threshold. For smooth non-polynomial activations, Theorem 5 establishes super-polynomial growth in scale-invariant RKHS cost for arbitrary fixed sequences of candidate priors. Together, these results provide meaningful generalizations of existing membership analyses and clarify limitations of exact neural-feature constructions for SVGPs.

Critical weaknesses

1. Limited development of the practical implications.

Given the paper’s primary area, "probabilistic methods," I believe its contribution should be assessed not only by its theoretical rigor but also by the methodological insights and practical guidance. Despite the meaningful theoretical results, their implications for practical modeling and inference decisions remain insufficiently developed.

Section 5.1 explicitly connects RKHS cost to the mean-dependent KL penalty, while Proposition 7 provides a useful extension to random network means. However, the paper does not quantify whether the asymptotic barriers imply substantial penalties at practically relevant dimensions, or how these limitations change when approximate representations are allowed.

A simple simulation or worked example examining how dimension, kernel choice, and approximation accuracy affect RKHS cost and the ELBO would help translate the theoretical results into concrete methodological guidance.

2. Insufficiently accessible presentation.

The overall progression is coherent, but abbreviations are not always expanded when first used, some notation is overloaded, and key definitions are deferred or insufficiently explained in the main text. These presentation issues make the analysis harder to follow.

  • NNGP is used in the abstract and introduction but expanded only in Section 3.2.
  • The underlying probability space Ω\Omega in L2(Ω)L^2(\Omega) is not introduced locally.
  • μ\mu is reused for both the prior mean function and an integration measure.
  • Eqs. (7)–(9) move from a scalar GP to multi-output neural-network layers without specifying the matrix dimensions or explaining how the scalar construction extends.
  • Eq (12) introduces HT\mathcal H_T as an operator range, but its norm—the quantity underlying the paper’s central cost—is explicitly defined only in Appendix A.2.

These are presentation issues rather than mathematical errors. Clearer definitions and a small worked example would make the analysis more accessible.

Minor comment.

Section 2.3 and Section 3 share the same title “RKHS Membership of Neurons.” Distinct titles would help separate the related-work discussion from the paper’s new results.

Questions for authors
  1. Could the authors provide a notation table in the appendix and clarify definitions and matrix dimensions at their first occurrence in the main text, particularly the examples identified in Weakness 2? This would improve consistency and make the analysis easier to follow.

  2. Proposition 7 establishes asymptotic, super-polynomial growth in scale-invariant RKHS cost. For commonly used smooth activations such as Softplus or tanh, could the authors provide activation-specific bounds or remarks clarifying when this growth becomes substantial at finite dimensions?

  3. Could the authors provide a numerical example using a concrete kernel to show how RKHS cost and the ELBO change with input dimension? This would help clarify when the theoretical cost barriers become relevant to practical inference.

3 comments

Thank you for your review! We appreciate you recognising the analytical framework and the generality of the results. You raised two main issues: underdeveloped practical implications, and presentation that was hard to follow in places. We hope the revision addresses both. The general response summarises the new experiments, so here we focus on your specific points.

Presentation (Question 1 and Weakness 2)

We have added a notation appendix (Appendix A), with tables for the theory and for the numerical methods of Appendix H, and fixed each issue you identified.

  • NNGP not defined early enough. It is now expanded in the abstract and introduced informally in the introduction as the covariance kernel of an infinitely wide network, with the formal definition in Section 3.2.
  • Probability space not introduced. (Ω,F,P)(\Omega,\mathcal F,\mathbb P) is now introduced where it is first used (Section 2.2).
  • μ\mu used twice. μ\mu now denotes only the prior mean. The measure in the interdomain integral formula is now υ\upsilon.
  • Moving from scalar to multi-output layers. Section 2.1 now explains that a multi-output layer is a set of independent scalar GPs sharing one kernel, and Equation (6) collects their variational means into the weight matrix Bℓ∈RMℓ×dℓB_\ell\in\mathbb R^{M_\ell\times d_\ell}.
  • RKHS norm deferred to the appendix. HT\mathcal H_T and its norm are now defined in Section 3.1, with a short explanation of why we use this form.
  • Duplicate section titles. Section 2.3 is now "Neural Inducing Features and RKHS Membership", distinct from Section 3.

We have also improved the proof overviews in the appendices.

Practical implications (Weakness 1)

The new Section 6 and Appendix H show how large the costs are at realistic dimensions, how they affect the ELBO, and how much approximation helps. Approximation can be much cheaper, especially with a tailored kernel.

When does the growth become substantial? (Question 2)

The growth in Proposition 7 comes from Theorem 5, so we made that result explicit at finite dimensions. The new Proposition 16 (Appendix H.4) restates the bound from Theorem 5's proof at any fixed dimension dd as Pr⁡{CT≤B}≤Fd,n(η)+Bη ad,nmd,n,\Pr\{C_T\le B\}\le F_{d,n}(\eta)+\frac{B}{\eta\,a_{d,n}m_{d,n}}, where Fd,nF_{d,n} depends only on the activation's degree-nn Hermite coefficient. In words, it limits how many neurons can be cheap under any fixed kernel.

Table 3 evaluates this for sigmoid, tanh⁡\tanh and Softplus at d=8,16,32,64d=8,16,32,64, giving a cost that at least 90% of neurons exceed under every fixed kernel.

  • Up to d=16d=16, the bound is weak and says little.
  • At d=32d=32, it becomes substantial for tanh⁡\tanh (more than 235).
  • At d=64d=64, it is substantial for all three activations (7.8×1047.8\times10^4 for sigmoid, 3.5×10133.5\times10^{13} for tanh⁡\tanh, 2.4×1032.4\times10^3 for Softplus).

These are not universal thresholds, as they depend on the parameter distributions, but they show that the asymptotic growth already matters at moderate dimensions.

A concrete kernel (Question 3)

Section 6 uses the "square-root NNGP prior", which by Theorem 2 and Corollary 3 has the lowest average cost of any fixed kernel, so it is a favourable benchmark rather than a deliberately bad choice. Weights are N(0,Id/d)\mathcal N(0,I_d/d) and biases N(0,1)\mathcal N(0,1). Projecting onto Hermite polynomials up to degree 32 gives lower bounds on the true costs (Appendix H.3).

RKHS cost (Figure 1, left, median lower bounds).

Activationd=16d=16d=32d=32
Sigmoid28.12.66×1042.66 \times 10^4
tanh⁡\tanh3.57×1043.57 \times 10^43.68×1093.68\times 10^9
Softplus54.05.11×1035.11\times 10^3

ELBO (Figure 1, right). For the single-neuron regression of the general response, with 512 data points, the table gives upper bounds on the typical best output weight (true value 1).

Activationd=12d=12d=16d=16d=24d=24d=32d=32
Sigmoid0.990.970.680.05
tanh⁡\tanh0.610.06<10−3<10^{-3}<10−3<10^{-3}
Softplus0.990.990.930.44

As these are upper bounds, the true optimal weights are even smaller. More data delays the effect but does not prevent it. So in this setting, the cost visibly affects inference from around d=12d=12 for tanh⁡\tanh and d=24d=24 for sigmoid and Softplus, and the correct neuron is essentially switched off a few dimensions later.

Fitted hyperparameters (Appendix H.7, Figure 7). With prior and noise variances fitted by exact GP inference and 2,048 observations, the learned amplitude still falls as dimension grows, to 0.19 for Softplus and 0.07 for cosine at d=40d=40. The exact GP still beats a pure-noise model in every trial, and a model that knows the neuron fits it equally well at every dimension. The attained ELBOs (Figure 8) mainly reflect the restriction to one inducing variable.

We hope these changes address your concerns, and we would be grateful if you would take them into account in your assessment!

Thank you for the detailed response and revisions. After reading the revised manuscript, I find that the finite-dimensional results, numerical examples, and presentation improvements substantially address my original concerns.

The added experiments clarify the implications for the ELBO in controlled single-neuron settings, although the broader implications for practical inference remain less fully explored.

Taking these improvements and the remaining limitations into account, I am raising my rating to weak accept.

Thank you for your careful reading of the revised work and for raising your score! We are glad to hear the revisions address most of your original concerns, and we agree that fuller study of the implications of these results in larger-scale variational inference problems is a natural direction for future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response - Summary of added experiments

General Response

We thank all three reviewers for their helpful comments.

To recap, our paper asks when a sparse Gaussian process can reproduce the forward pass of a deep neural network, using neurons such as σ(w⊤x+b)\sigma(w^\top x + b) as inducing features. This raises two questions.

  1. Can a neuron be used at all? It must lie in the prior's RKHS (Theorems 1 and 4).
  2. If so, what does it cost? Its RKHS norm is a penalty in the ELBO (Proposition 6), and for typical neurons this penalty grows superpolynomially with dimension (Theorem 5, Proposition 7 and Theorem 8).

These results are asymptotic, and all reviewers asked how large the costs are at practical dimensions. We have therefore added a new Section 6, with details and further experiments in the new Appendix H, measuring these costs and their effect on inference. Code for all experiments is available at an anonymous repository. We summarise the main results here.

Setup. We consider sigmoid, tanh⁡\tanh and Softplus neurons with randomly drawn weights and biases, and measure them using the scale-invariant "relative" RKHS cost used throughout the paper, CT(θ)=tr(T)∥σθ∥HT2C_T(\theta) = \text{tr}(T) \|\sigma_\theta\|^2_{\mathcal H_T}.

Exact representation is expensive at relevant dimensions, even when using the best kernel

We first use the "square-root NNGP prior", which Theorem 2 and Corollary 3 identify as giving the lowest average cost of any fixed kernel. Even so, costs grow very quickly with dimension, as Theorem 5 predicts. From d=16d=16 to d=32d=32, median costs rise from 28 to about 27,000 for sigmoid, from 54 to about 5,100 for Softplus, and from about 36,000 to 3.7 billion for tanh⁡\tanh (Figure 1, left). For cosine activations, closed-form costs show this growth analytically (Appendix H.1).

No fixed kernel avoids the obstruction in high dimensions

The new Proposition 16 (Appendix H.4) applies the bound behind Theorem 5 at any fixed dimension and to every kernel, bounding the cost exceeded by a given proportion of neurons. The bound is weak in low dimensions, but at d=64d=64 it shows that, under any fixed kernel, at least 90% of neurons cost more than 7.80×1047.80\times 10^4 (sigmoid), 3.46×10133.46 \times 10^{13} (tanh⁡\tanh) and 2.36×1032.36 \times 10^3 (Softplus) (Table 3). For tanh⁡\tanh it is already substantial at d=32d=32.

The RKHS cost changes what the ELBO prefers

We generate regression data from a single known neuron and let the ELBO choose only its output weight, whose true value is 1. For Softplus with 512 data points, the typical best weight is close to 1 at d=16d=16, at most 0.44 at d=32d=32 and below 10−510^{-5} at d=64d=64 (Figure 1, right). Even when the correct neuron is known, the ELBO can prefer to switch it off because it is too expensive under the prior.

Fitted hyperparameters (new Appendix H.7). We repeat this experiment with prior and noise variances fitted by exact GP inference, using 2,048 observations so that the exact GP still beats a pure-noise model in every trial up to d=40d=40. At given hyperparameters the optimal amplitude does not depend on the posterior covariance, and it is still suppressed as the cost grows, falling to 0.07 for cosine and 0.19 for Softplus at d=40d=40 (Figure 7). A model that knows the neuron fits it equally well at every dimension, so this is not due to a lack of information in the data. We also report attained ELBOs (Figure 8), but these mainly reflect our restriction to a single inducing variable, which leaves most of the prior variance unexplained, rather than the RKHS cost.

Approximate activations can be cheaper

We agree that approximate representations matter. Our obstructions concern exact non-polynomial neurons, so approximating them, for example by polynomials, can certainly reduce costs. We illustrate one way, replacing each neuron with a low-degree polynomial under a kernel designed for it. At d=32d=32, average costs fall to about 2.5 for sigmoid, 50 for Softplus and 670 for tanh⁡\tanh, with errors of 6.5%, 1.8% and 8.0%. Under any fixed kernel, average exact costs there are at least 1.5×1051.5\times10^5, 2.3×1042.3\times10^4 and 8.6×1098.6\times10^9 (Appendix H.5). This kernel covers only a finite-dimensional space, however. Many other approximations are possible, which may also require changing the kernel, and a fuller study of them is future work.

Takeaways

Our results concern the cost of reproducing a given network exactly, under a kernel that is not tailored to that network. They do not show that trained deep GPs must fail, as an optimiser that is free to choose its posterior mean can avoid expensive neurons. Exact activation-shaped neurons are affordable in low dimensions, while in higher dimensions approximation is the more promising route.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.