- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper shows how continuous BNNs can be discretized and ranked on a validation dataset for achieving higher quality of uncertainty while also drastically reducing the number of computations. The authors propose a ranking process for the case where the true predictive distribution of the validation dataset is known as well as for the case where only the samples from the true distribution are given, as is common in real-life datasets.
The effectiveness is shown by applying the proposed method to Monte Carlo Dropout, Hypermodels, Variational Neural Networks, Deep Ensembles, Layer Ensembles and Bayes By Backpropagation.
Critical strengths
A good idea to characterize uncertainty is proposed. Specifically, this paper proposed a method to discretize Bayesian Neural Networks and rank their samples to create deterministic models that achieve higher uncertainty quality and require fewer computations. The authors formally derive the ranking process of Layer Ensembles and extend it to other BNNs and datasets that do not have uncertainty targets for the validation dataset and show the effectiveness of both ranking methods.
Critical weaknesses
The proposed method is not applicable to current large models, since the quantity of the parameters and hyperparameters are so huge. That is to say, in practice, it is difficult to scale. And therefore it is difficult to provide state-of-the-art performance for practical uses.
Questions for authors
For each input example x, the best set of parameters and hyperparameters should be optimized separately. Is this understanding right?
In addition, Figure 1 needs more explanations. For example, how is it generated, and what distributions are they in terms of equations (4) and (5)?
3 comments
thanks for the authors' rebuttal
Thanks for the efforts to provide the rebuttal. However, currently no experimental results are provided on the concerned large models. My major concern was not resolved.
I will maintain my score.
Thank you for your comment.
We added Table 5 to the paper that analyzes the inference speed differences of classical, VNN and BBB versions of a ResNet-152 network ranked on validation images. We implement inference optimizations for VNNs that utilize parallel inference of multiple samples in each tensor operation, and we leave BBB with a usual implementation to showcase the best and the worst scenarios for BNNs and DiBNNs, depending on the available hardware constraints.
We thank the reviewer for expressing practical concerns about the method. We address them in the following.
The proposed method is not applicable to current large models
The cost of the ranking process is a multiplication of the model's inference cost, but it is done once after training and requires T×∣Dv∣ model inferences. For huge models such as LLMs, the parameter-efficient BNN alternatives may be only twice larger (for models such as VNNs, BBB, MCD, LayerEnsembles), so a "7b" BNN model will still fit in a consumer GPU. The inference cost of a single such BNN is usually smaller than ×2 due to parallelization of operations, and the total Monte Carlo inference is the main bottleneck as it requires multiple model runs per output. Following our ranking process, we can reduce this issue of BNNs by selecting only a small number of samples, and the count will be selected based on the available computational budget. Additionally, when only a portion of the model is made Bayesian, the issue becomes much easier to solve, but at the cost of the uncertainty quality.
Per-input optimization
The ranking is global and is done for the whole dataset (on a validation subset), so the selected samples are estimated to be best for the whole set of data points, with the expectation to extrapolate on the unseen data the same way as the model is expected to perform well on unseen data for its regular predictions.
Figure 1 needs more explanations
Figure 1 is a simplified example of the problem for a single x value where we mark the model's predictive distribution with a purple line and label it "p" and mark the true distribution with a green line and label it "q". The distributions are created by hand and visualize that the same points can be used to estimate two different distributions. The actual ranking does not require exact intersections of distributions but is based on the idea that a worse approximation of the predictive distribution p(y∣x) that is created using fewer approximation points will be a closer approximation of the true distribution q(y∣x).