Discrete Bayesian Neural Networks

Bayesian Neural Networks (BNNs) are used for uncertainty estimation by considering a usually continuous probability distribution over the parameters of a neural network. In this paper, we show how continuous BNNs can be discretized and ranked on a validation dataset for achieving higher quality of uncertainty while also drastically reducing the number of computations. We propose a ranking process for the case where the true predictive distribution of the validation dataset is known as well as for the case where only the samples from the true distribution are given, as is common in real-life datasets. We evaluate the proposed Discrete Bayesian Neural Networks (DiBNNs) versions of Variational Neural Networks, Hypermodels, Deep Ensembles, Bayes By Backpropagation, Monte Carlo Dropout and Layer Ensembles on The Neural Testbed which provides true uncertainty for the test set. When the uncertainty targets are given for the validation dataset, DiBNNs achieve up to $2.1$ times lower KL-divergence values and require an order of magnitude fewer computations than BNNs. If the validation dataset does not have uncertainty targets, DiBNNs achieve up to $2.6$ times better KL-divergence values at lower samples, which is beneficial under a limited computational budget.