acceptodds
Under review as a conference paper at ICLR 2027

Function-Space Priors for Bayesian Neural Networks via Anchor Matching

Abstract

Bayesian neural networks typically place priors on weights, although inductive properties such as smoothness, periodicity, and geometric structure are more naturally specified in function space. Gaussian processes (GPs) provide a standard way to encode such properties, but matching a finite neural-network prior to a GP over entire functions can lead to infinite KL divergence. We instead match the distributions at a finite set of input locations, or anchors. We formulate anchor matching as finding a weight prior whose induced anchor distribution matches the GP while minimizing KL divergence from a base weight prior, and derive the unique solution under finite-KL conditions. Since the network-induced anchor distribution is implicit, we estimate a bounded log-density ratio from base-prior and GP samples, and use it to construct a reweighted prior with adjustable strength. We then use PAC-Bayes bounds to select the reweighting strength and training temperature over a pre-specified finite grid while preserving the risk guarantee up to a logarithmic grid-size penalty. We further characterize conditions for finite-cost anchor matching, show that exact matching can yield infinite KL divergence from Gaussian posteriors, and derive the optimal bounded reweighting and its effect on the PAC-Bayes complexity term. Overall, our framework provides a principled way to transfer structured function-space information into network weight priors while retaining compatibility with posterior inference and finite-sample risk guarantees.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian neural networks, function-space priors, Gaussian processes, prior matching, PAC-Bayes

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_f3yaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The submission proposes incorporating Gaussian process (GP) information into Bayesian neural network weight priors by matching function values at a finite set of anchor inputs. It derives the minimum-KL modification of a base prior that achieves exact matching while preserving the conditional distribution of weights given anchor outputs. For practical implementation, it learns a clipped quadratic approximation to the target-to-base log density ratio from simulated outputs and uses it to reweight the prior. The authors show that exact matching is incompatible with Gaussian posteriors, and characterize the optimal bounded reweighting. Reweighting strength and training temperature are selected by a PAC-Bayes bound. In small-scale experiments, the authors report closer agreement with GP posteriors, NLL gains, and nonvacuous bounds.

Critical strengths
  • The paper is well written and the argument is easy to follow.
  • The construction of anchor matching via prior reweighting seems like a sensible idea and estimating the log density ratio from fixed samples means the resulting prior has a tractable unnormalized density that standard inference methods can use directly.
  • The method also preserves the conditional weight distribution at the anchor points which leaves this as a separate modeling choice to the base prior.
  • Theorem 3 is an interesting and useful negative result.
Critical weaknesses
  • By Proposition 1, the lifted prior keeps the base prior conditional distribution given the anchor outputs. Function behaviour between and away from anchors is therefore inherited from the base prior. The introduction motivates the method with smoothness, periodicity and geometric structure, but:
    • every experiment uses a stationary target, with no periodic or otherwise structured kernel.
    • every experiment uses M = 12 anchors, with no ablation on M or on anchor placement, even for up to 8-dimensional UCI inputs in the regression experiment.
    • there is no discussion of how the method scales with M and the method itself seems very computationally heavy from the start.
  • The submission includes only a very small number of references in its main methodology sections but many of the considerations and techniques have closely related work that would deserve to be referenced here, but only shows up in the related literature section, such as Kessler et al.
  • The most important GP functional prior baselines for this method that are even mentioned in the related work section, namely, Tran et al. and Flam-Shepherd et al., are missing completely from the experiments.

I ran the code that was submitted as an anonymous repository and found several issues, that I will list below sometimes together with issues from the main text:

  • The reported Wasserstein-2 distance in Table 1 seems to be wrong, as the paper states what is referred to in the code as w2_second_moment instead of w2_mean_covariance which I guess makes the proposed method gain an advantage over the baseline methods as the mean shift is disregarded.
  • The experiment in Table 2 does not make a lot of sense as the FSVI baseline methods are fitted to a different objective and on top use mean-field variational inference which is known to perform much worse than NUTS due to a very restrictive posterior approximation. Moreover, when looking at the code, fSVI and GFSVI do not seem to be implemented correctly as they only differ by the use of a regularization term and do not correspond to the explanation in Appendix C.3.
  • In Section 4.1, GP-density weighting is implemented without clipping (T_E = 200 does not appear in the code), and its optimum is at lambda = 0, i.e. the base prior, so the comparison does not make sense.
  • Section 4.2 uses a setup that heavily favors the proposed method as the base prior scales are arbitrarily set to 2 and 4, which results in functions that are heavily overdispersed compared to the ground-truth GP that generated the data and no tuned base prior results are reported. Also, the target is a GP and no misspecified setting is tested.
  • In the UCI regression experiments, the target GP seems to be fitted on the validation set that is used to select lambda. Also, requiring R-hat values in function space to be less than 1.01 and otherwise rerunning chains could bias the posterior variance downward, as finding functionally diverse modes during MCMC sampling is discouraged this way.
Questions for authors
  1. Can the authors clarify their contributions over the works mentioned in the related literature section?
  2. Which dataset and which run does Figure 2 show? Rerunning the configured protocol on dataset 0, I obtain a base-prior D_GP of 0.418, matching the stated 0.41. The anchor prior gives 0.398 rather than 0.22. (I used JAX as the reported pytorch version did not install for me and ran the chains sequentially, so the difference may partly reflect the software environment)
  3. In Table 3, which λ,βtrain\lambda, \beta_{train} and βbound\beta_{bound} did the PAC-Bayes rule select on each split, and how often was λ=0\lambda = 0? What bound does the base prior achieve under the same rule?
  4. Can the authors revise section 2.3 to describe the estimator actually run in C.1-C.3, with unclipped fitting and post hoc, data-dependent clipping? Alternatively, can those experiments be rerun with clipping inside the objective at a fixed level?
  5. How does the anchor-matching method behave when the target GP differs from the data-generating process, for example through a wrong lengthscale or kernel?
  6. In Table 3, are the baseline methods correctly implemented and used, especially, as the function-space VI methods are only evaluated at the anchor points?
  7. Given the substantial computational overhead, what is the reason that one should use anchor-matching over a) an actual GP, or b) a ReLU BNN with isotropic standard Gaussian priors paired with an efficient MCMC inference method in the settings considered or additional settings?
11 comments
AuthorsOpenReview ↗

Response to Reviewer f3ya (1/6): periodic target, anchor count and placement (W1, W2)

We thank the reviewer for the careful reading and for running the released code. The detailed comments led to concrete corrections and to new experiments, notably the periodic target and anchor placement, that make the scope of the method clearer. Our findings for each issue are below; all corrections are in the revision, which marks changes in blue and adds the new experiments in Appendix D. All numbers in our replies come from one rerun of every method on one machine with fixed seeds, so that old and new methods share the same setup.

As the reviewer notes, by Proposition 1 the behavior between and away from the anchors comes from the base prior. We tested this directly.

W1 (Periodic target)

Anchor matching outperforms the base prior and even exact regression with a non-periodic GP, but it remains far from the periodic GP, because the shape between anchors comes from the base prior. Both the data and the target GP use a periodic kernel (period 2, lengthscale 1); anchors are 0.73 apart. NUTS, 10 simulated datasets; DGPD_{\mathrm{GP}} and SD ratio are relative to the periodic GP posterior:

Methodgap NLL, mean (median)DGPD_{\mathrm{GP}}gap SD ratio
Base prior2.156 (2.04)2.1615.8
Anchor matching, M=12M = 121.746 (1.57)1.249.5
Exact GP, periodic (correct)−0.11301
Exact GP, Matérn-5/2 (no periodic structure)2.303 (1.79)1.30

Anchor matching beats the base prior in DGPD_{\mathrm{GP}} on 10/10 datasets and in gap NLL on 7/10, and its mean gap NLL is lower than exact regression with a non-periodic GP (1.75 vs. 2.30). The prior carries the periodic correlations between anchors: each anchor inside the gap has an anchor outside it at correlation 0.85, so data outside the gap inform the anchors inside it.

It remains far from the periodic GP, because the shape between anchors comes from the ReLU base prior. This is the limitation the reviewer identifies. M=24M = 24 (datasets 0–4) lowers gap NLL from 2.01 to 1.65 and DGPD_{\mathrm{GP}} from 1.34 to 1.23, but the posterior stays much wider than the periodic GP's. The revision adds this experiment in Appendix D.5 and states the scope there: anchor matching transfers the target's structure at the anchors, while structure between anchors must come from denser anchors or from the base prior.

W2 (Anchor count and placement)

Anchor matching improves on the base prior for every anchor count from 6 to 48, with gains up to about 24 anchors; placement matters only in that the anchors must cover the region without data. Section 4.2 setting, width 32, NUTS, five simulated datasets (base prior: gap NLL 1.302, DGPD_{\mathrm{GP}} 0.579, NUTS 27 s). Wall-clock times are on a 2-core CPU, averaged over three datasets.

Anchor count:

MMgap NLLDGPD_{\mathrm{GP}}better than base (NLL, DGPD_{\mathrm{GP}})prior construction timeNUTS inference time
61.0700.3345/5, 5/564 s33 s
121.0080.2614/5, 5/564 s34 s
240.9810.1704/5, 5/566 s38 s
481.0320.1834/5, 5/573 s44 s

DGPD_{\mathrm{GP}} falls from 0.33 to 0.17 as MM grows and saturates around 24 anchors. Across widths 32 to 96, it stays at 0.26 (W3).

Anchor placement (M=12M = 12):

Placementgap NLLDGPD_{\mathrm{GP}}better than base (NLL, DGPD_{\mathrm{GP}})
Even grid (paper)1.008 (0.083)0.261 (0.040)4/5, 5/5
Uniformly random1.053 (0.159)0.330 (0.073)4/5, 5/5
Outside the gap only1.390 (0.239)0.622 (0.105)0/5, 0/5
Inside the gap only1.055 (0.171)0.128 (0.011)3/5, 5/5

Random placement does about as well as the grid. Anchors placed only where data exist do nothing, because the likelihood already determines the function there; anchors in the gap give the lowest DGPD_{\mathrm{GP}}. The rule is that anchors must cover the region without data; beyond that, their exact positions matter little. The revision states this rule.

AuthorsOpenReview ↗

Response to Reviewer f3ya (2/6): computational cost, references, missing baselines (W3–W5)

W3 (Computational cost)

The added cost is small: about one minute of data-free prior construction, nearly independent of the network size, and 20–30% more NUTS time.

Network width (M=12M = 12, same setting):

Width (parameters)base: gap NLL / DGPD_{\mathrm{GP}}anchor matching: gap NLL / DGPD_{\mathrm{GP}}better than base (NLL, DGPD_{\mathrm{GP}})prior construction timeNUTS inference time: base / anchor
32 (1,153)1.302 / 0.5791.008 / 0.2614/5, 5/564 s27 / 34 s
64 (4,353)1.332 / 0.6330.912 / 0.2625/5, 5/568 s80 / 104 s
96 (9,601)1.387 / 0.6520.947 / 0.2625/5, 5/574 s182 / 222 s

Prior construction takes 64–74 s for all MM and widths we tried; for 8× more weights it grows by about 15%, because the density ratio is fitted on the MM-dimensional anchor outputs. NUTS takes 20–30% longer than with the base prior. For comparison, fSVI takes 39 s per dataset, and fitting the GPi-G prior with our implementation takes 6,640 s on the same machine.

W4 (References in the methodology sections)

We agree that the methodology sections should cite this work where it is used, not only in the related work. The construction in Proposition 1 is the marginally specified prior of Kessler et al. (2015). The revision cites it, with Poole & Raftery (2000) and the Markov melding work of Goudie et al. (2019), in Section 2.2 where Proposition 1 is stated. It also cites the Gaussian density penalties of Rudner et al. (2023) and Kozyrskiy et al. (2023) where GP-density weighting is introduced in Section 4.1. Section 2.5 now cites the PAC-Bayes selection literature (Catoni, 2007; Dziugaite & Roy, 2017; Thiemann et al., 2017; Pérez-Ortiz et al., 2021; Lotfi et al., 2022).

W5 (Missing baselines: Tran et al. and Flam-Shepherd et al.)

Both are now included in the Section 4.2 setting, with the same NUTS protocol as the other priors. Under this protocol, anchor matching on p0p_0 outperforms both: GPi-G is overconfident in the gap, and NUTS does not mix under the prior of Flam-Shepherd et al.

  • GPi-G (Tran et al., 2022): layer-wise weight and bias scales fitted by Wasserstein-1 matching to the GP.
  • Flam-Shepherd et al. (2017): a mean and a scale per weight, fitted by minimizing the KL from the network's function-space law (moment-matched Gaussian) to the GP over random input sets (20,000 Adam steps, 9 minutes, done once). We ran it with and without anchor matching on top.

20 simulated datasets, mean (s.e.):

Weight priorAnchor matchinggap NLLDGPD_{\mathrm{GP}}gap SD ratio
GPi-G (Tran et al., 2022)no5.307 (1.478)0.735 (0.049)0.35
Flam-Shepherd et al. (2017)no2.030 (0.346)0.522 (0.046)1.01
Flam-Shepherd et al. (2017)yes2.133 (0.364)0.552 (0.047)0.98
p0p_0 (ours)yes1.088 (0.104)0.255 (0.032)1.01
Exact GP1.031 (0.093)01

GPi-G. Its posterior is overconfident in the gap (SD ratio 0.35). Anchor matching on our base prior p0p_0 has lower DGPD_{\mathrm{GP}} on 19/20 datasets and lower gap NLL on 14/20.

Flam-Shepherd et al. A priori, the fitted prior matches the GP closely: the KL on a 200-point grid falls from 56.9 to 0.81, and the prior SD in the gap matches the GP's (ratio 1.00). The fit gets there by placing 70% of the weight means more than two scales from zero, with output-layer scales around 0.003, and the resulting posterior is strongly multimodal. NUTS does not mix under it on any of the 20 datasets, even after the longer rerun (largest R^\hat{R} per run 2.5–11, ESS below 3). We sample the standardized weights (non-centered); step size and acceptance are normal, but each chain stays in its own mode. Anchor matching has almost nothing to correct on top of this prior (anchor KL 0.01 nats). Under the NUTS protocol used for every prior, anchor matching on p0p_0 has lower gap NLL on 16/20 datasets and lower DGPD_{\mathrm{GP}} on 17/20.

Together, these show two ways a fitted Gaussian weight prior falls short here. Fitted on scales alone (GPi-G), it is overconfident in the gap. Fitted with means as well (Flam-Shepherd), it matches the GP a priori, but NUTS does not mix under it. Anchor matching on p0p_0 avoids both. Appendix D.8 also reports a zero-mean variant of Flam-Shepherd et al. (scales only), with and without anchor matching, and the convergence diagnostics of every run.

AuthorsOpenReview ↗

Response to Reviewer f3ya (3/6): code-level issues: Table 1, fSVI/GFSVI, Section 4.1 (W6–W8)

W6 (Table 1, W2W_2 statistic)

The reviewer is right. The anchor-matching row reported the second-moment W2W_2 (0.179); the mean–covariance W2W_2 described in the text is 0.280 [0.196, 0.366]. The baselines are unchanged (0.643 [0.558, 0.684] and 0.653 [0.562, 0.698]), so the ranking stands. Table 1 is corrected.

W7 (fSVI/GFSVI, Appendix C.3)

The reviewer is right that the code did not match Appendix C.3. Both methods evaluate the linearized function-space KL at the MM anchors as context points. In UCI, GFSVI also uses the regularized KL of Cinquin & Bamler (γ\gamma selected on calibration data) and a linearized expected log-likelihood, while fSVI uses a Monte Carlo likelihood; in the synthetic experiment, they differ only through the regularization. Appendix C.3 now says exactly this. We also reran the synthetic baselines with 12 fresh uniform context points on [−4, 4] at every step, as in the original methods; anchor matching with NUTS still outperforms both:

Method (20 simulated datasets)gap NLL, mean (median)DGPD_{\mathrm{GP}}
fSVI, anchors as contexts5.59 (1.92)0.702
fSVI, fresh contexts5.81 (2.41)0.589
GFSVI, anchors as contexts7.62 (4.72)0.740
GFSVI, fresh contexts7.12 (3.67)0.692
Anchor matching + NUTS1.09 (0.98)0.255

Fresh contexts slightly improve DGPD_{\mathrm{GP}}, and gap NLL changes little. Anchor matching still beats fresh-context fSVI on 17/20 datasets in gap NLL and 19/20 in DGPD_{\mathrm{GP}}. In UCI, all function-space methods keep the anchors as contexts, so every method receives GP information at the same inputs.

W8 (Section 4.1, TE=200T_E = 200)

The reviewer is right that the clip was not implemented. With the clip added to the GP log-density score, it binds on 0 of 50,000 base draws (the most negative score is −21.8), so the curve is identical and the optimum is λ=0\lambda = 0 in all five repetitions.

λ=0\lambda = 0 is the point of this baseline, not an artifact. GP-density weighting has the same form as the Gaussian density penalties on network evaluations in prior work (Rudner et al., 2023; Kozyrskiy et al., 2023; Section 5). In the Gaussian setting of Section 4.1, it changes the anchor precision to K0−1+λKA−1K_0^{-1} + \lambda K_A^{-1}, which shrinks the covariance in every direction. It therefore cannot reach a target that is wider than the base prior in some direction, and its best choice is to leave the base prior unchanged. Multiplying by the density ratio (the minimum-KL lift) accounts for the anchor distribution the base prior already induces and reaches W2W_2 = 0.280 against 0.653. This is what Section 4.1 is designed to show; the revision says so explicitly and removes the inactive TET_E.

Reviewer_Z4rXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes a method to construct weights prior that match GP function values at finite set of input locations (anchors) while minimizing KL divergence from a base prior. The authors study when exact matching is possible and when it conflicts with Gaussian posterior inference. In addition, the authors constructed a bounded density-ratio approximation makes the construction practical. Finally, the paper introduced PAC-Bayes bounds that select the reweighting strength and training temperature. The authors conducted experiments on synthetic examples and small regression datasets.

Critical strengths
  • The paper is well-motivated. The authors have interesting results on the feasibility and ReLU activation incompatibility that go beyond the classical minimum-KL marginal replacement construction. From that, the authors explain why closer GP matching can increase posterior-to-prior KL and motivate bounded reweighting.
  • The authors demonstrated the effectiveness of the proposed reweighted priors over the base priors using NUTs. The reduction in mean discrepancy from GP posterior supports this effectiveness.
Critical weaknesses
  • The empirical evaluation is limited to synthetic examples and three small UCI regression datasets with simple neural networks. It does not demonstrate whether density-ratio estimation remain effective and computationally practical with larger network, higher-dimensional inputs, or larger anchor sets.
  • The evidence for an advantage over existing function-space methods is limited. Figure 2 changes the inference procedure as well as the prior. The minimum-KL construction and finite-family PAC-Bayes selection also build on established principles. The more specific theoretical contributions are informative, but the experiments do not yet show a sufficient practical benefit.
Questions for authors
  • How do matching accuracy, runtime, and prediction change with network size and anchor count?
  • Does the advantage over function-space baselines remains with comparable inference procedures and computational budgets?
7 comments
AuthorsOpenReview ↗

Response to Reviewer Z4rX (1/2): matching accuracy and runtime versus network width and anchor count

We thank the reviewer for the focused questions and answer both with new experiments. The revised PDF marks changes in blue, and the new experiments are in Appendix D. All numbers below come from one rerun of every method on one machine with fixed seeds, so that old and new methods share the same setup. Where the setup is unchanged, they are close to the submission (mean DGPD_{\mathrm{GP}} in Section 4.2: 0.563 vs. 0.55 for the base prior, 0.255 vs. 0.26 for anchor matching).

W1, Q1 (Matching accuracy, runtime and prediction versus network size and anchor count)

Anchor matching remains effective as the network and the anchor set grow: it is closer to the GP than the base prior at every width and anchor count we tried (lower DGPD_{\mathrm{GP}} on all datasets), and prior construction takes about one minute regardless of the number of weights.

Section 4.2 setting (no training inputs in [−1.5, 1.5], "the gap"), NUTS, five simulated datasets (each a new draw from the GP with 64 noisy observations). Wall-clock times are on a 2-core CPU, averaged over three datasets. Matching accuracy is the sliced 2-Wasserstein distance (SW2SW_2) between the prior's anchor outputs and the GP marginal at the anchors; it is also the criterion that selects λ\lambda, with no observed data (Appendix C.2).

Anchor count (width 32; base prior: gap NLL 1.302, DGPD_{\mathrm{GP}} 0.579, NUTS 27 s):

MMgap NLLDGPD_{\mathrm{GP}}SW2SW_2 at the anchors (p0p_0 → selected λ\lambda)better than base (NLL, DGPD_{\mathrm{GP}})prior construction timeNUTS inference time
61.0700.33413.0 → 0.575/5, 5/564 s33 s
121.0080.26112.0 → 0.364/5, 5/564 s34 s
240.9810.17011.6 → 0.314/5, 5/566 s38 s
481.0320.18311.4 → 0.314/5, 5/573 s44 s

Network width (M=12M = 12):

Width (parameters)base: gap NLL / DGPD_{\mathrm{GP}}anchor matching: gap NLL / DGPD_{\mathrm{GP}}better than base (NLL, DGPD_{\mathrm{GP}})prior construction timeNUTS inference time: base / ours
32 (1,153)1.302 / 0.5791.008 / 0.2614/5, 5/564 s27 / 34 s
64 (4,353)1.332 / 0.6330.912 / 0.2625/5, 5/568 s80 / 104 s
96 (9,601)1.387 / 0.6520.947 / 0.2625/5, 5/574 s182 / 222 s
  • Anchors. DGPD_{\mathrm{GP}} falls from 0.33 to 0.17 as MM grows and saturates around 24 anchors.
  • Width. The base prior drifts away from the GP as the network grows (DGPD_{\mathrm{GP}} 0.58 → 0.65), while anchor matching stays at 0.26.
  • Runtime. Prior construction takes about one minute and grows from 64 s to 74 s for 8× more parameters. The density ratio is fitted on the MM-dimensional anchor outputs, so this step does not depend on the number of weights. At M=12M = 12, anchor matching adds 20–30% to the NUTS time; at M=48M = 48, it adds 64%.
  • Larger networks. Since construction time grew by only about 15% for 8× more weights, the cost for larger networks is dominated by the inference method. The anchor-matched prior is an unnormalized weight-space density, so it can be used with any scalable sampler or variational method.
  • Input dimension. The density ratio is fitted on the MM anchor outputs, so the input dimension enters mainly through the number of anchors needed to cover the region without data (anchor placement results are in our reply to f3ya, W2). Table 2 already uses 8-dimensional inputs (Concrete, Energy) with the same construction, and the table above shows the cost of more anchors: prior construction grows from 64 s to 73 s from 6 to 48 anchors.
AuthorsOpenReview ↗

Response to Reviewer Z4rX (2/2): comparable inference and compute

W2, Q2 (Does the advantage remain with comparable inference and compute?)

We agree that Figure 2 changes the inference as well as the prior. A new experiment (Appendix D.1) varies one at a time. With the same inference, the advantage over weight-space priors remains: with NUTS, anchor matching outperforms the base prior and GPi-G, the Gaussian weight prior that Tran et al. (2022) fit to the GP by Wasserstein matching (Appendix D.1). Against fSVI, anchor matching with NUTS reaches a much lower gap NLL (1.09 vs. 5.59) at about 2.5× the wall-clock time, but within the mean-field family fSVI is better. Mean-field VI uses the same family and settings for every method as for fSVI/GFSVI in Appendix C.2 (20,000 Adam steps, eight likelihood particles per step). Section 4.2 setting, 20 simulated datasets, mean (s.e.); the gap SD ratio is the posterior SD of f(x)f(x) divided by the exact GP's, averaged over the gap (1 matches the GP):

Prior / methodNUTS: gap NLLNUTS: DGPD_{\mathrm{GP}}mean-field VI: gap NLLmean-field VI: DGPD_{\mathrm{GP}}mean-field VI: gap SD ratio
Base prior1.160 (0.082)0.563 (0.086)10.288 (2.116)0.874 (0.044)0.16
GPi-G (Tran et al., 2022)5.307 (1.478)0.735 (0.049)10.675 (2.013)0.905 (0.059)0.20
Anchor matching1.088 (0.104)0.255 (0.032)9.712 (1.925)0.858 (0.044)0.17
fSVI——5.592 (1.863)0.702 (0.033)0.28
fSVI, fresh contexts per step——5.813 (1.758)0.589 (0.033)0.36
GFSVI——7.618 (1.863)0.740 (0.031)0.21
Exact GP (reference)1.031 (0.093)0

Same inference. With NUTS fixed, anchor matching has lower DGPD_{\mathrm{GP}} than the base prior on 20/20 datasets and than GPi-G on 19/20. With the prior fixed, gap NLL goes from 9.71 (mean-field VI) to 1.09 (NUTS).

Same variational family. Here fSVI is better: anchor matching wins on 2/20 datasets. Mean-field weight-space posteriors stay at gap SD ratio 0.16–0.20 under all three priors we tested, so most of the GP information in the weight-space prior does not reach the gap posterior; fSVI partly avoids this because its objective acts in function space.

Compute. Per dataset, anchor matching with NUTS takes about 100 s (65 s prior construction + 35 s NUTS), about 2.5× the 39 s of fSVI (GFSVI 38 s, mean-field VI with the base prior 13 s). It reaches gap NLL 1.09 against 5.59 for fSVI (5.81 with fresh contexts; exact GP 1.03), and beats fresh-context fSVI on 17/20 datasets in gap NLL and 19/20 in DGPD_{\mathrm{GP}}. This combination needs a prior density over weights; fSVI places the GP in the variational objective and has no weight-space prior to sample from.

UCI. All methods in Table 2 share the same rank-10 Gaussian family and splits.

W2 (The evidence for an advantage over existing function-space methods is limited)

The new comparisons sharpen the claim. Against GPi-G, the advantage is clear under identical inference (lower DGPD_{\mathrm{GP}} on 19/20 datasets). Against function-space VI, anchor matching with NUTS is better, at about 2.5× the wall-clock time of fSVI. Within the mean-field family, fSVI is better. The robustness results support the same conclusion: misspecified target GP, anchor placement, a periodic target, and an isotropic N(0,I)\mathcal{N}(0, I) base prior (reply to f3ya). Appendix D reports all of these comparisons.

Reviewer_Z4rXOpenReview ↗

Official Comment by Reviewer Z4rX

Thank the authors for the additional experiments and clarifications. The matched-inference comparisons address part of my concern, and the width and anchor-count studies provide useful information about computational cost. However, several concerns remain.

I share reviewer f3ya's concern about the GPi-G baseline. The latest response provides fitting diagnostics and a tuned-base-prior comparison, but the explanation for GPi-G’s poor performance depends on the chosen ReLU architecture. Have the authors compared both methods using tanh, as used in the original GPi-G regression experiments, with the same data and inference procedure? Please also clarify whether the prior-fitting settings match the original implementation. A plateau in the critic objective does not by itself establish that the prior fit is adequate.

I also agree that the inconsistent numbers and strength of the claims needed correction. The authors acknowledge that the manuscript mixed original runs with reruns and reported a different Wasserstein statistic from the one described. I appreciate these corrections. The revised tables, figures, and text should consistently identify the experiment and evaluation protocol used. The claims should also reflect the limitations revealed by the rebuttal: matching at finite anchors does not establish recovery of the full GP function law, the estimation guarantee does not cover the experimental fits, and most of the improvement in Table 3 comes from the selection rule rather than anchor matching.

Finally, the empirical scope remains limited. Most new experiments extend the same one-dimensional synthetic setting, while the real-data evaluation remains restricted to small and low-dimensional UCI regression datasets. The width study reaches fewer than 10,000 parameters, so it does not establish scalability to substantially larger networks. Under matched mean-field inference, fSVI also performs better, which limits the claimed practical advantage.

The rebuttal improves the paper and addresses several specific questions. Nevertheless, the remaining uncertainty about the baseline comparison and the limited evidence beyond these settings leave me unconvinced about the overall contribution. I believe that the paper needs a major revision to fix the issues and to be fully evaluated. Therefore, I would keep my current rating for the current form.

Reviewer_t1sBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper constructs BNN weight priors by matching a GP distribution at selected inputs. It derives the smallest change to a base prior in KL divergence, introduces bounded reweighting to keep posterior KL finite, and uses PAC-Bayes bounds to select hyperparameters. Experiments evaluate prior matching, posterior predictions, and regression performance.

Critical strengths
  1. The result showing that exact GP matching can cause infinite KL from Gaussian posteriors is useful. It explains why bounded, approximate matching may be preferable even when exact matching is possible.

  2. The paper connects the prior construction to posterior inference and risk guarantees. The guarantees account for both hyperparameter selection and Monte Carlo estimation, making the numerical bounds more credible.

Critical weaknesses
  1. The gap experiment compares NUTS with mean-field variational inference, making it difficult to separate improvements from the prior and the inference method. The UCI evaluation uses only three runs and lacks a direct comparison with existing GP-to-BNN prior-learning methods.

  2. Table 3 reports bounds on clipped squared loss over a population that includes training data. Held-out performance is not reported for the selected models, so these results do not establish whether selecting by the bound improves predictions on unseen data.

Questions for authors
  1. Could you compare with a GP-to-BNN prior-learning method, such as GPi-G [1], cited in your related work, using the same inference method and data splits? I would like to understand whether the proposed prior construction improves on existing approaches.

  2. How well do the models selected in Table 3 perform on held-out data? Are the reported bounds tighter than simply using the observed training losses and assuming the maximum possible loss at every point in T?

[1] Tran, B.H., Rossi, S., Milios, D. and Filippone, M., 2022. All you need is a good functional prior for Bayesian deep learning. Journal of Machine Learning Research, 23(74), pp.1-56.

4 comments
AuthorsOpenReview ↗

Response to Reviewer t1sB (1/2): comparison with GPi-G under the same inference

We thank the reviewer for the two concrete questions, which we answer with new experiments. The revised PDF marks changes in blue, and the new experiments are in Appendix D. All numbers below come from one rerun of every method on one machine with fixed seeds, so that old and new methods share the same setup. Where the setup is unchanged, they are close to the submission (mean DGPD_{\mathrm{GP}} in Section 4.2: 0.563 vs. 0.55 for the base prior, 0.255 vs. 0.26 for anchor matching; Table 3 is unchanged).

W1, Q1 (Comparison with GPi-G under the same inference)

Under the same NUTS sampler, anchor matching outperforms GPi-G in both gap NLL and DGPD_{\mathrm{GP}} and is close to the exact GP, whereas GPi-G is overconfident in the gap (table below).

We implemented GPi-G following Tran et al. (2022). In each layer ℓ\ell, weights follow N(0,sℓ2/dℓ)\mathcal{N}(0, s_\ell^2/d_\ell) and biases N(0,tℓ2)\mathcal{N}(0, t_\ell^2), where dℓd_\ell is the layer's input width. The layer-wise scales sℓ,tℓs_\ell, t_\ell are fitted by Wasserstein-1 matching to the target GP on 200 measurement points (200-200 softplus critic, gradient penalty 10, reference optimizers). The fitted prior goes into the same NUTS sampler, data and seeds as the other priors.

Section 4.2 setting (no training inputs in the unobserved interval [−1.5, 1.5], "the gap"), NUTS, 20 simulated datasets (each a new draw from the GP with 64 noisy observations, as in Section 4.2), mean (s.e.). The gap SD ratio is the posterior SD of f(x)f(x) divided by the exact GP's, averaged over the gap (Appendix C.2); 1 matches the GP, and below 1 is overconfident:

Prior (all NUTS)gap NLLDGPD_{\mathrm{GP}}gap SD ratio (GP = 1)
Base prior p0p_01.160 (0.082)0.563 (0.086)1.37
GPi-G5.307 (1.478); median 3.370.735 (0.049)0.35
Anchor matching1.088 (0.104); median 0.980.255 (0.032)1.01
Exact GP (reference)1.031 (0.093)01

Anchor matching has lower DGPD_{\mathrm{GP}} than GPi-G on 19/20 datasets and lower gap NLL on 14/20; against the base prior, DGPD_{\mathrm{GP}} is lower on 20/20. GPi-G is fitted to the GP's function-space law over the whole domain, but its posterior is overconfident in the gap: its SD is about one third of the GP's. Anchor matching corrects the prior law at the anchors and keeps the base prior's conditional law elsewhere (Proposition 1).

Anchor matching on top of GPi-G selects λ=0\lambda = 0 on all 20 datasets. The lifting cost KL(γ∥ν0)\mathrm{KL}(\gamma \Vert \nu_0) of Proposition 1 is about 139 nats from GPi-G versus about 5 nats from our base prior p0p_0. p0p_0 is deliberately wide (its prior SD in the gap is about 11 times the GP's), so it covers the GP marginal and the lift is cheap.

AuthorsOpenReview ↗

Response to Reviewer t1sB (2/2): separating prior and inference; held-out results and the trivial bound for Table 3

W1 (The gap experiment compares NUTS with mean-field VI)

We agree that this comparison confounds the prior and the inference method. A new experiment (Appendix D.1) varies one at a time. With NUTS, anchor matching is the best of the three priors and close to the exact GP; with mean-field VI, all weight-space priors are overconfident in the gap, and fSVI is better. Mean-field VI uses the same family and settings for every method as for fSVI/GFSVI in Appendix C.2 (20,000 Adam steps, eight likelihood particles per step). 20 datasets, mean (s.e.):

Prior / methodNUTS: gap NLLNUTS: DGPD_{\mathrm{GP}}mean-field VI: gap NLLmean-field VI: DGPD_{\mathrm{GP}}mean-field VI: gap SD ratio
Base prior1.160 (0.082)0.563 (0.086)10.288 (2.116)0.874 (0.044)0.16
GPi-G5.307 (1.478)0.735 (0.049)10.675 (2.013)0.905 (0.059)0.20
Anchor matching1.088 (0.104)0.255 (0.032)9.712 (1.925)0.858 (0.044)0.17
fSVI——5.592 (1.863)0.702 (0.033)0.28
fSVI, fresh contexts per step——5.813 (1.758)0.589 (0.033)0.36
GFSVI——7.618 (1.863)0.740 (0.031)0.21
Exact GP (reference)1.031 (0.093)0
  • Prior effect (NUTS column): anchor matching is best and close to the exact GP.
  • Inference effect (same row): anchor matching goes from 9.71 under mean-field VI to 1.09 under NUTS.
  • Within mean-field VI: anchor matching improves on the base prior (15/20 datasets), but fSVI is better (anchor matching wins 2/20). The weight-space posteriors stay at gap SD ratio 0.16–0.20 under all three priors, so most of the prior's gap information does not reach the posterior. fSVI partly avoids this because its objective acts in function space.

The anchor-matched prior is a weight-space density, so it works with any weight-space sampler, and its GP information survives when the inference can represent the posterior. fSVI has no weight-space prior density and cannot be combined with NUTS.

W2, Q2 (Held-out performance of the models selected in Table 3, and the trivial bound)

Selecting by the bound matches or slightly improves on validation-based selection on held-out data, without using validation data. Each selected model is evaluated on the remainder TT, which is never used for fitting, selection or certification. Mean over seven splits, NLL s.e. in parentheses:

Selection ruleDiabetes boundheld-out riskheld-out NLLYacht boundheld-out riskheld-out NLL
PAC-Bayes bound0.4320.06675.439 (0.010)0.8780.00101.472 (0.109)
Validation NLL0.5300.06695.458 (0.019)0.9420.00101.519 (0.091)
Training risk2.2340.08156.055 (0.182)3.7800.00072.767 (0.397)

On the trivial bound (nrS+∣T∣)/(n+∣T∣)(n r_S + |T|)/(n + |T|), the reviewer is right. With ∣T∣/n≈0.30|T|/n \approx 0.30, it is tighter than our certificate on all 14 splits: 0.285 vs. 0.432 (Diabetes) and 0.238 vs. 0.878 (Yacht). Section 4.3 now states this, Appendix D.10 reports the trivial bound for every selection rule, and we remove the emphasis on nonvacuousness. Two facts put this in context:

  1. As a selection rule, the trivial bound is monotone in training risk, so it selects the training-risk minimizer (third row). On Diabetes, that model is worse on held-out risk (0.082 vs. 0.067) and NLL (6.05 vs. 5.44). On Yacht, its held-out squared error is slightly lower, but it is overconfident (NLL 2.77 vs. 1.47). The certificate accounts for the complexity of the selected posterior; the trivial bound cannot.
  2. Population size. The trivial bound is tight here only because the unseen part of the population is small; it tends to 1 as ∣T∣/n|T|/n grows. Our certificate does not depend on ∣T∣|T|: by Corollary A3, the same value also bounds the expected loss when the observations are i.i.d., where the trivial bound is 1. Appendix D.10 gives, for each split, the ratio above which the trivial bound becomes the looser one.

Thank you for the rebuttal and the new experiments. My questions overall have been answered, and I am keeping my score of 3.

We sincerely thank the reviewer for carefully considering our rebuttal and additional experiments, and for confirming that our responses have addressed your questions.

Your constructive feedback was particularly valuable in helping us strengthen the experimental evaluation and clarify the scope of our conclusions. In particular, the matched-inference comparisons and the additional held-out evaluations for PAC-Bayes selection have allowed us to present a more careful assessment of both the strengths and limitations of our approach.

We have incorporated these results and clarifications into the revised manuscript, and we greatly appreciate the time and care you have devoted to reviewing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.