- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The submission proposes incorporating Gaussian process (GP) information into Bayesian neural network weight priors by matching function values at a finite set of anchor inputs. It derives the minimum-KL modification of a base prior that achieves exact matching while preserving the conditional distribution of weights given anchor outputs. For practical implementation, it learns a clipped quadratic approximation to the target-to-base log density ratio from simulated outputs and uses it to reweight the prior. The authors show that exact matching is incompatible with Gaussian posteriors, and characterize the optimal bounded reweighting. Reweighting strength and training temperature are selected by a PAC-Bayes bound. In small-scale experiments, the authors report closer agreement with GP posteriors, NLL gains, and nonvacuous bounds.
Critical strengths
- The paper is well written and the argument is easy to follow.
- The construction of anchor matching via prior reweighting seems like a sensible idea and estimating the log density ratio from fixed samples means the resulting prior has a tractable unnormalized density that standard inference methods can use directly.
- The method also preserves the conditional weight distribution at the anchor points which leaves this as a separate modeling choice to the base prior.
- Theorem 3 is an interesting and useful negative result.
Critical weaknesses
- By Proposition 1, the lifted prior keeps the base prior conditional distribution given the anchor outputs. Function behaviour between and away from anchors is therefore inherited from the base prior. The introduction motivates the method with smoothness, periodicity and geometric structure, but:
- every experiment uses a stationary target, with no periodic or otherwise structured kernel.
- every experiment uses M = 12 anchors, with no ablation on M or on anchor placement, even for up to 8-dimensional UCI inputs in the regression experiment.
- there is no discussion of how the method scales with M and the method itself seems very computationally heavy from the start.
- The submission includes only a very small number of references in its main methodology sections but many of the considerations and techniques have closely related work that would deserve to be referenced here, but only shows up in the related literature section, such as Kessler et al.
- The most important GP functional prior baselines for this method that are even mentioned in the related work section, namely, Tran et al. and Flam-Shepherd et al., are missing completely from the experiments.
I ran the code that was submitted as an anonymous repository and found several issues, that I will list below sometimes together with issues from the main text:
- The reported Wasserstein-2 distance in Table 1 seems to be wrong, as the paper states what is referred to in the code as
w2_second_momentinstead ofw2_mean_covariancewhich I guess makes the proposed method gain an advantage over the baseline methods as the mean shift is disregarded. - The experiment in Table 2 does not make a lot of sense as the FSVI baseline methods are fitted to a different objective and on top use mean-field variational inference which is known to perform much worse than NUTS due to a very restrictive posterior approximation. Moreover, when looking at the code, fSVI and GFSVI do not seem to be implemented correctly as they only differ by the use of a regularization term and do not correspond to the explanation in Appendix C.3.
- In Section 4.1, GP-density weighting is implemented without clipping (T_E = 200 does not appear in the code), and its optimum is at lambda = 0, i.e. the base prior, so the comparison does not make sense.
- Section 4.2 uses a setup that heavily favors the proposed method as the base prior scales are arbitrarily set to 2 and 4, which results in functions that are heavily overdispersed compared to the ground-truth GP that generated the data and no tuned base prior results are reported. Also, the target is a GP and no misspecified setting is tested.
- In the UCI regression experiments, the target GP seems to be fitted on the validation set that is used to select lambda. Also, requiring R-hat values in function space to be less than 1.01 and otherwise rerunning chains could bias the posterior variance downward, as finding functionally diverse modes during MCMC sampling is discouraged this way.
Questions for authors
- Can the authors clarify their contributions over the works mentioned in the related literature section?
- Which dataset and which run does Figure 2 show? Rerunning the configured protocol on dataset 0, I obtain a base-prior D_GP of 0.418, matching the stated 0.41. The anchor prior gives 0.398 rather than 0.22. (I used JAX as the reported pytorch version did not install for me and ran the chains sequentially, so the difference may partly reflect the software environment)
- In Table 3, which and did the PAC-Bayes rule select on each split, and how often was ? What bound does the base prior achieve under the same rule?
- Can the authors revise section 2.3 to describe the estimator actually run in C.1-C.3, with unclipped fitting and post hoc, data-dependent clipping? Alternatively, can those experiments be rerun with clipping inside the objective at a fixed level?
- How does the anchor-matching method behave when the target GP differs from the data-generating process, for example through a wrong lengthscale or kernel?
- In Table 3, are the baseline methods correctly implemented and used, especially, as the function-space VI methods are only evaluated at the anchor points?
- Given the substantial computational overhead, what is the reason that one should use anchor-matching over a) an actual GP, or b) a ReLU BNN with isotropic standard Gaussian priors paired with an efficient MCMC inference method in the settings considered or additional settings?
11 comments
Response to Reviewer f3ya (2/6): computational cost, references, missing baselines (W3–W5)
W3 (Computational cost)
The added cost is small: about one minute of data-free prior construction, nearly independent of the network size, and 20–30% more NUTS time.
Network width (, same setting):
| Width (parameters) | base: gap NLL / | anchor matching: gap NLL / | better than base (NLL, ) | prior construction time | NUTS inference time: base / anchor |
|---|---|---|---|---|---|
| 32 (1,153) | 1.302 / 0.579 | 1.008 / 0.261 | 4/5, 5/5 | 64 s | 27 / 34 s |
| 64 (4,353) | 1.332 / 0.633 | 0.912 / 0.262 | 5/5, 5/5 | 68 s | 80 / 104 s |
| 96 (9,601) | 1.387 / 0.652 | 0.947 / 0.262 | 5/5, 5/5 | 74 s | 182 / 222 s |
Prior construction takes 64–74 s for all and widths we tried; for 8× more weights it grows by about 15%, because the density ratio is fitted on the -dimensional anchor outputs. NUTS takes 20–30% longer than with the base prior. For comparison, fSVI takes 39 s per dataset, and fitting the GPi-G prior with our implementation takes 6,640 s on the same machine.
W4 (References in the methodology sections)
We agree that the methodology sections should cite this work where it is used, not only in the related work. The construction in Proposition 1 is the marginally specified prior of Kessler et al. (2015). The revision cites it, with Poole & Raftery (2000) and the Markov melding work of Goudie et al. (2019), in Section 2.2 where Proposition 1 is stated. It also cites the Gaussian density penalties of Rudner et al. (2023) and Kozyrskiy et al. (2023) where GP-density weighting is introduced in Section 4.1. Section 2.5 now cites the PAC-Bayes selection literature (Catoni, 2007; Dziugaite & Roy, 2017; Thiemann et al., 2017; Pérez-Ortiz et al., 2021; Lotfi et al., 2022).
W5 (Missing baselines: Tran et al. and Flam-Shepherd et al.)
Both are now included in the Section 4.2 setting, with the same NUTS protocol as the other priors. Under this protocol, anchor matching on outperforms both: GPi-G is overconfident in the gap, and NUTS does not mix under the prior of Flam-Shepherd et al.
- GPi-G (Tran et al., 2022): layer-wise weight and bias scales fitted by Wasserstein-1 matching to the GP.
- Flam-Shepherd et al. (2017): a mean and a scale per weight, fitted by minimizing the KL from the network's function-space law (moment-matched Gaussian) to the GP over random input sets (20,000 Adam steps, 9 minutes, done once). We ran it with and without anchor matching on top.
20 simulated datasets, mean (s.e.):
| Weight prior | Anchor matching | gap NLL | gap SD ratio | |
|---|---|---|---|---|
| GPi-G (Tran et al., 2022) | no | 5.307 (1.478) | 0.735 (0.049) | 0.35 |
| Flam-Shepherd et al. (2017) | no | 2.030 (0.346) | 0.522 (0.046) | 1.01 |
| Flam-Shepherd et al. (2017) | yes | 2.133 (0.364) | 0.552 (0.047) | 0.98 |
| (ours) | yes | 1.088 (0.104) | 0.255 (0.032) | 1.01 |
| Exact GP | 1.031 (0.093) | 0 | 1 |
GPi-G. Its posterior is overconfident in the gap (SD ratio 0.35). Anchor matching on our base prior has lower on 19/20 datasets and lower gap NLL on 14/20.
Flam-Shepherd et al. A priori, the fitted prior matches the GP closely: the KL on a 200-point grid falls from 56.9 to 0.81, and the prior SD in the gap matches the GP's (ratio 1.00). The fit gets there by placing 70% of the weight means more than two scales from zero, with output-layer scales around 0.003, and the resulting posterior is strongly multimodal. NUTS does not mix under it on any of the 20 datasets, even after the longer rerun (largest per run 2.5–11, ESS below 3). We sample the standardized weights (non-centered); step size and acceptance are normal, but each chain stays in its own mode. Anchor matching has almost nothing to correct on top of this prior (anchor KL 0.01 nats). Under the NUTS protocol used for every prior, anchor matching on has lower gap NLL on 16/20 datasets and lower on 17/20.
Together, these show two ways a fitted Gaussian weight prior falls short here. Fitted on scales alone (GPi-G), it is overconfident in the gap. Fitted with means as well (Flam-Shepherd), it matches the GP a priori, but NUTS does not mix under it. Anchor matching on avoids both. Appendix D.8 also reports a zero-mean variant of Flam-Shepherd et al. (scales only), with and without anchor matching, and the convergence diagnostics of every run.
Response to Reviewer f3ya (3/6): code-level issues: Table 1, fSVI/GFSVI, Section 4.1 (W6–W8)
W6 (Table 1, statistic)
The reviewer is right. The anchor-matching row reported the second-moment (0.179); the mean–covariance described in the text is 0.280 [0.196, 0.366]. The baselines are unchanged (0.643 [0.558, 0.684] and 0.653 [0.562, 0.698]), so the ranking stands. Table 1 is corrected.
W7 (fSVI/GFSVI, Appendix C.3)
The reviewer is right that the code did not match Appendix C.3. Both methods evaluate the linearized function-space KL at the anchors as context points. In UCI, GFSVI also uses the regularized KL of Cinquin & Bamler ( selected on calibration data) and a linearized expected log-likelihood, while fSVI uses a Monte Carlo likelihood; in the synthetic experiment, they differ only through the regularization. Appendix C.3 now says exactly this. We also reran the synthetic baselines with 12 fresh uniform context points on [−4, 4] at every step, as in the original methods; anchor matching with NUTS still outperforms both:
| Method (20 simulated datasets) | gap NLL, mean (median) | |
|---|---|---|
| fSVI, anchors as contexts | 5.59 (1.92) | 0.702 |
| fSVI, fresh contexts | 5.81 (2.41) | 0.589 |
| GFSVI, anchors as contexts | 7.62 (4.72) | 0.740 |
| GFSVI, fresh contexts | 7.12 (3.67) | 0.692 |
| Anchor matching + NUTS | 1.09 (0.98) | 0.255 |
Fresh contexts slightly improve , and gap NLL changes little. Anchor matching still beats fresh-context fSVI on 17/20 datasets in gap NLL and 19/20 in . In UCI, all function-space methods keep the anchors as contexts, so every method receives GP information at the same inputs.
W8 (Section 4.1, )
The reviewer is right that the clip was not implemented. With the clip added to the GP log-density score, it binds on 0 of 50,000 base draws (the most negative score is −21.8), so the curve is identical and the optimum is in all five repetitions.
is the point of this baseline, not an artifact. GP-density weighting has the same form as the Gaussian density penalties on network evaluations in prior work (Rudner et al., 2023; Kozyrskiy et al., 2023; Section 5). In the Gaussian setting of Section 4.1, it changes the anchor precision to , which shrinks the covariance in every direction. It therefore cannot reach a target that is wider than the base prior in some direction, and its best choice is to leave the base prior unchanged. Multiplying by the density ratio (the minimum-KL lift) accounts for the anchor distribution the base prior already induces and reaches = 0.280 against 0.653. This is what Section 4.1 is designed to show; the revision says so explicitly and removes the inactive .
Response to Reviewer f3ya (1/6): periodic target, anchor count and placement (W1, W2)
We thank the reviewer for the careful reading and for running the released code. The detailed comments led to concrete corrections and to new experiments, notably the periodic target and anchor placement, that make the scope of the method clearer. Our findings for each issue are below; all corrections are in the revision, which marks changes in blue and adds the new experiments in Appendix D. All numbers in our replies come from one rerun of every method on one machine with fixed seeds, so that old and new methods share the same setup.
As the reviewer notes, by Proposition 1 the behavior between and away from the anchors comes from the base prior. We tested this directly.
W1 (Periodic target)
Anchor matching outperforms the base prior and even exact regression with a non-periodic GP, but it remains far from the periodic GP, because the shape between anchors comes from the base prior. Both the data and the target GP use a periodic kernel (period 2, lengthscale 1); anchors are 0.73 apart. NUTS, 10 simulated datasets; DGP and SD ratio are relative to the periodic GP posterior:
Anchor matching beats the base prior in DGP on 10/10 datasets and in gap NLL on 7/10, and its mean gap NLL is lower than exact regression with a non-periodic GP (1.75 vs. 2.30). The prior carries the periodic correlations between anchors: each anchor inside the gap has an anchor outside it at correlation 0.85, so data outside the gap inform the anchors inside it.
It remains far from the periodic GP, because the shape between anchors comes from the ReLU base prior. This is the limitation the reviewer identifies. M=24 (datasets 0–4) lowers gap NLL from 2.01 to 1.65 and DGP from 1.34 to 1.23, but the posterior stays much wider than the periodic GP's. The revision adds this experiment in Appendix D.5 and states the scope there: anchor matching transfers the target's structure at the anchors, while structure between anchors must come from denser anchors or from the base prior.
W2 (Anchor count and placement)
Anchor matching improves on the base prior for every anchor count from 6 to 48, with gains up to about 24 anchors; placement matters only in that the anchors must cover the region without data. Section 4.2 setting, width 32, NUTS, five simulated datasets (base prior: gap NLL 1.302, DGP 0.579, NUTS 27 s). Wall-clock times are on a 2-core CPU, averaged over three datasets.
Anchor count:
DGP falls from 0.33 to 0.17 as M grows and saturates around 24 anchors. Across widths 32 to 96, it stays at 0.26 (W3).
Anchor placement (M=12):
Random placement does about as well as the grid. Anchors placed only where data exist do nothing, because the likelihood already determines the function there; anchors in the gap give the lowest DGP. The rule is that anchors must cover the region without data; beyond that, their exact positions matter little. The revision states this rule.