- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies whether standard neural-network neurons can be used exactly as inducing features in sparse variational GPs. This requires the neurons to belong to the GP prior’s RKHS.
Under the stated assumptions, a neuron belongs to its own single-layer NNGP RKHS only when its activation is polynomial. The paper then considers other kernels. It proves non-membership results for activations with derivative jumps and shows that smooth non-polynomial neurons have very large RKHS costs as dimension increases. It extends the cost result to random networks and derives a consequence for the Gaussian-regression ELBO.
Critical strengths
-
The paper considers general candidate kernels.
Theorem 4 is more interesting to me than a non-membership result for one particular neural kernel. Under its assumptions, changing to another fixed trace-class covariance does not solve the problem for a positive-measure set of neurons.
-
The RKHS cost is connected to the variational objective.
The ELBO result gives a concrete reason to care about the norm. Under the stated budget, the gain in data fit cannot pay for the cost of exactly reproducing a typical random network mean. The network result also checks that cancellation between neurons does not invalidate this argument.
Critical weaknesses
I did not find a critical flaw in the mathematical results. I have two smaller concerns:
-
The size of the cost at finite dimensions is unclear. Theorem 5 gives an asymptotic result, but I had trouble judging how large the cost would actually be for a standard activation. Equation (94) provides a bound, though it still depends on the distribution of the Hermite coefficients. A worked example for tanh or Softplus would help show when the bound becomes informative and how much it tells us at realistic dimensions.
-
FMGP [1] is missing from the related work. As far as I understood FMGPs, it also uses the RKHS formulation of SVGPs to work with prescribed neural-network means, so it seems directly relevant. Its approximation argument on compact sets addresses a different question from exact RKHS membership. Discussing that difference would help explain how the results here relate to an existing construction, especially since FMGP keeps the mean fixed while fitting uncertainty.
Neither point gives me a strong reason to reject the paper, but both would make its practical relevance easier to assess.
[1] Luis A. Ortega, Simón Rodríguez-Santana, and Daniel Hernández-Lobato. Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learning. arXiv:2412.04177, 2024.
Questions for authors
-
Could you work out the bound in Equation (94) for tanh or Softplus? At roughly what dimensions does it become informative? This would help me judge how large the representation cost is in practice.
-
Could you discuss the connection to FMGP? Both papers use the RKHS formulation of SVGPs, but FMGP uses approximation on compact sets and keeps the mean fixed while fitting uncertainty. Which conclusions here apply to that setting?
3 comments
Thank you for the detailed response and the additional experiments. The finite-dimensional results and the discussion of FMGP address my questions, and I maintain my positive assessment.
Thank you for your engagement throughout the discussion period! We are glad our revisions successfully address your questions, and we appreciate your positive assessment.
Thank you very much for your review! We greatly appreciate your recognition of the generality of the theoretical results and the connection to variational inference. You raised two points, namely how large the cost is at finite dimensions and the missing comparison with FMGP. We hope to have addressed both in the revision. The new experiments are summarised in the general response, so here we focus on your specific questions.
Working out the bound for tanh and Softplus (Question 1)
The bound you refer to as Equation (94) is Equation (92) in the revision. We have made it an explicit result. The new Proposition 16 (Appendix H.4) states that, at any fixed dimension d and for every fixed kernel T, Pr{CT≤B}≤Fd,n(η)+ηad,nmd,nB,Fd,n(η)=Pr{cn(Rd,b)2≤η}. As you noted, it depends on the distribution of the Hermite coefficients cn. We estimate these distributions numerically, then choose the degree n≤48 and threshold η that give the strongest bound. We use 16,384 parameter draws with a 99% confidence allowance that holds across all reported cases. Inputs are N(0,Id), weights N(0,Id/d) and biases N(0,1).
Result (Table 3). The table gives a cost that at least 90% of neurons exceed, under every fixed kernel.
When is it informative? The table should be compared with an elementary bound. Every eigenvalue of a fixed kernel is at most its trace, so every neuron satisfies CT(θ)≥∥σθ∥L22. The 10th percentile of this is about 0.36 for tanh and 0.18 for Softplus. Proposition 16 therefore tells us nothing new up to d=16, becomes informative and substantial for tanh at d=32, and becomes informative for Softplus at d=64, where both bounds are large.
Appendix H.4 now states this comparison explicitly. We do not claim these are exact crossover dimensions, since they depend on the weight and bias distributions.
A sharper picture for a specific kernel. A bound that holds for every kernel is necessarily conservative. Under the square-root NNGP prior, which has the lowest average cost of any fixed kernel, median costs are already large at moderate dimensions. At d=32, the median cost is about 5,100 for Softplus and 3.7 billion for tanh (Table 2, Figure 1). Section 6 also shows the effect on the ELBO, as summarised in the general response.
Relation to FMGP (Question 2)
Thank you for pointing us to this paper, which is indeed very relevant! We now cite this work in Section 2.4. We see the two papers as complementary.
Our conclusions relate to the FMGP setting as follows.
In this sense, FMGP fits naturally with our conclusions. Rather than paying for an expensive mean inside the GP, it takes the network's mean as given and focuses on uncertainty.
Thank you again for your helpful suggestions!