Function-Space Priors for Bayesian Neural Networks via Anchor Matching

Bayesian neural networks typically place priors on weights, although inductive properties such as smoothness, periodicity, and geometric structure are more naturally specified in function space. Gaussian processes (GPs) provide a standard way to encode such properties, but matching a finite neural-network prior to a GP over entire functions can lead to infinite KL divergence. We instead match the distributions at a finite set of input locations, or anchors. We formulate anchor matching as finding a weight prior whose induced anchor distribution matches the GP while minimizing KL divergence from a base weight prior, and derive the unique solution under finite-KL conditions. Since the network-induced anchor distribution is implicit, we estimate a bounded log-density ratio from base-prior and GP samples, and use it to construct a reweighted prior with adjustable strength. We then use PAC-Bayes bounds to select the reweighting strength and training temperature over a pre-specified finite grid while preserving the risk guarantee up to a logarithmic grid-size penalty. We further characterize conditions for finite-cost anchor matching, show that exact matching can yield infinite KL divergence from Gaussian posteriors, and derive the optimal bounded reweighting and its effect on the PAC-Bayes complexity term. Overall, our framework provides a principled way to transfer structured function-space information into network weight priors while retaining compatibility with posterior inference and finite-sample risk guarantees.