acceptodds
Under review as a conference paper at ICLR 2027

Discrete Bayesian Neural Networks

Abstract

Bayesian Neural Networks (BNNs) are used for uncertainty estimation by considering a usually continuous probability distribution over the parameters of a neural network. In this paper, we show how continuous BNNs can be discretized and ranked on a validation dataset for achieving higher quality of uncertainty while also drastically reducing the number of computations. We propose a ranking process for the case where the true predictive distribution of the validation dataset is known as well as for the case where only the samples from the true distribution are given, as is common in real-life datasets. We evaluate the proposed Discrete Bayesian Neural Networks (DiBNNs) versions of Variational Neural Networks, Hypermodels, Deep Ensembles, Bayes By Backpropagation, Monte Carlo Dropout and Layer Ensembles on The Neural Testbed which provides true uncertainty for the test set. When the uncertainty targets are given for the validation dataset, DiBNNs achieve up to times lower KL-divergence values and require an order of magnitude fewer computations than BNNs. If the validation dataset does not have uncertainty targets, DiBNNs achieve up to times better KL-divergence values at lower samples, which is beneficial under a limited computational budget.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
bayesian neural networks, uncertainty estimation, machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dZTCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper shows how continuous BNNs can be discretized and ranked on a validation dataset for achieving higher quality of uncertainty while also drastically reducing the number of computations. The authors propose a ranking process for the case where the true predictive distribution of the validation dataset is known as well as for the case where only the samples from the true distribution are given, as is common in real-life datasets.

The effectiveness is shown by applying the proposed method to Monte Carlo Dropout, Hypermodels, Variational Neural Networks, Deep Ensembles, Layer Ensembles and Bayes By Backpropagation.

Critical strengths

A good idea to characterize uncertainty is proposed. Specifically, this paper proposed a method to discretize Bayesian Neural Networks and rank their samples to create deterministic models that achieve higher uncertainty quality and require fewer computations. The authors formally derive the ranking process of Layer Ensembles and extend it to other BNNs and datasets that do not have uncertainty targets for the validation dataset and show the effectiveness of both ranking methods.

Critical weaknesses

The proposed method is not applicable to current large models, since the quantity of the parameters and hyperparameters are so huge. That is to say, in practice, it is difficult to scale. And therefore it is difficult to provide state-of-the-art performance for practical uses.

Questions for authors

For each input example x, the best set of parameters and hyperparameters should be optimized separately. Is this understanding right?

In addition, Figure 1 needs more explanations. For example, how is it generated, and what distributions are they in terms of equations (4) and (5)?

3 comments

We thank the reviewer for expressing practical concerns about the method. We address them in the following.

The proposed method is not applicable to current large models

The cost of the ranking process is a multiplication of the model's inference cost, but it is done once after training and requires T×∣Dv∣T \times |D_v| model inferences. For huge models such as LLMs, the parameter-efficient BNN alternatives may be only twice larger (for models such as VNNs, BBB, MCD, LayerEnsembles), so a "7b" BNN model will still fit in a consumer GPU. The inference cost of a single such BNN is usually smaller than ×2\times2 due to parallelization of operations, and the total Monte Carlo inference is the main bottleneck as it requires multiple model runs per output. Following our ranking process, we can reduce this issue of BNNs by selecting only a small number of samples, and the count will be selected based on the available computational budget. Additionally, when only a portion of the model is made Bayesian, the issue becomes much easier to solve, but at the cost of the uncertainty quality.

Per-input optimization

The ranking is global and is done for the whole dataset (on a validation subset), so the selected samples are estimated to be best for the whole set of data points, with the expectation to extrapolate on the unseen data the same way as the model is expected to perform well on unseen data for its regular predictions.

Figure 1 needs more explanations

Figure 1 is a simplified example of the problem for a single xx value where we mark the model's predictive distribution with a purple line and label it "p" and mark the true distribution with a green line and label it "q". The distributions are created by hand and visualize that the same points can be used to estimate two different distributions. The actual ranking does not require exact intersections of distributions but is based on the idea that a worse approximation of the predictive distribution p(y∣x)p(y|x) that is created using fewer approximation points will be a closer approximation of the true distribution q(y∣x)q(y|x).

Reviewer_dZTCOpenReview ↗

thanks for the authors' rebuttal

Thanks for the efforts to provide the rebuttal. However, currently no experimental results are provided on the concerned large models. My major concern was not resolved.

I will maintain my score.

Thank you for your comment.

We added Table 5 to the paper that analyzes the inference speed differences of classical, VNN and BBB versions of a ResNet-152 network ranked on 20,00020,000 validation images. We implement inference optimizations for VNNs that utilize parallel inference of multiple samples in each tensor operation, and we leave BBB with a usual implementation to showcase the best and the worst scenarios for BNNs and DiBNNs, depending on the available hardware constraints.

Reviewer_vwQKOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

The paper proposes Discrete Bayesian Neural Networks (DiBNNs), which construct finite, ordered subsets of model samples from trained uncertainty estimation methods. Samples are selected greedily using validation KL divergence when reference predictive distributions are available, or validation log-likelihood when only observed labels are available. Predictions are cached to avoid repeated network evaluations during selection. Experiments on the Neural Testbed cover six model families and assess predictive quality at different inference sample budgets. The results show gains with distribution-valued validation targets, while the benefits of likelihood-based ranking depend on the method and budget.

Critical strengths
  1. The paper addresses a practically relevant inference problem. Selecting a reusable subset of model samples can improve predictive quality at a fixed inference budget. Table 1 provides encouraging evidence: KL-ranked VNNs achieve a median KL of 0.17 with a median of 64 samples, compared with 0.36 for the 1,000-sample baseline. These results demonstrate the potential value of selection when suitable validation targets are available.

  2. The evaluation across six model families provides useful comparative evidence. The common testbed enables direct comparison against reference predictive distributions and reveals differences in how individual methods respond to selection. The paper also acknowledges limitations of likelihood-based ranking, including settings in which it does not improve performance.

Critical weaknesses
  1. The methodological contribution is incremental, and the mathematical motivation contains overstated claims.

    The greedy selection procedure explicitly builds on Layer Ensembles and Caruana et al. The main extensions are its application across additional model families, likelihood-based ranking, and cached evaluation. These are useful developments, but the mathematical treatment primarily formalizes the procedure: Eq. (6) defines the optimization target, Eq. (9) specifies a greedy approximation, and Eqs. (10)–(12) use the standard relationship between expected KL divergence and expected log-likelihood. There is little analysis of when selection improves predictive quality or when empirical likelihood ranking reliably identifies better subsets.

    Some motivational claims go beyond what is established. Lines 151–157 assert that sufficiently close predictive distributions admit identical finite-sample approximations. Closeness alone does not imply this: two binary models can have arbitrarily close predictive distributions while all their component probabilities remain on opposite sides of 0.5, preventing their finite averages from coinciding. Similarly, lines 175–182 do not establish why convergence toward a misspecified distribution guarantees an improving smaller subset.

    I do not consider a general optimality theorem necessary for this method. However, the distinction between empirical intuition and established conclusions needs to be clearer. Given the close relationship to existing selection methods, the paper would need stronger analytical insight or more compelling empirical evidence to establish the significance of its extensions.

  2. The empirical evidence supports useful but conditional improvements, and the likelihood-ranking results require clarification.

    The experiments provide a controlled evaluation on synthetic GP regression problems, but do not establish how broadly the benefits extend beyond this setting. Even within the testbed, improvements at matched budgets are method-dependent. For example, at 10 samples in Table 3, likelihood-ranked VNN improves median KL from 0.72 to 0.55, whereas likelihood-ranked BBB worsens it from 0.54 to 0.95. The authors acknowledge these differences; nevertheless, they limit the strength of the empirical contribution supporting a general extension across model families.

    Another concern about the interpretation of these experiments. Lines 353–354 state that point-wise validation labels are sampled from the corresponding reference distribution. In the released find_best_samples implementation, observation noise is added when generating likelihood-validation labels, but the KL reference variance does not receive the corresponding addition. This appears to make likelihood selection and KL evaluation target different distributions. Clarifying this discrepancy is necessary before attributing the observed behavior to the limitations of label-based ranking.

Questions for authors
  1. Scope of the mathematical claims. Lines 151–157 state that sufficiently close predictive distributions admit identical finite-sample approximations, while lines 175–182 argue that a smaller subset should improve upon the full sample set when the model predictive distribution differs from the reference distribution. Are these intended as general claims or illustrative intuition? Please state any additional assumptions needed for the claims, or qualify their scope accordingly.

  2. Consistency of the experimental target. Does q(y∣x)q(y\mid x) describe latent function values or noisy observations? In the released find_best_samples implementation, observation noise is added when generating labels for likelihood-based ranking, while the KL reference variance does not receive the corresponding addition. Please confirm whether this code generated the reported tables and explain how the label-generating distribution and the KL reference correspond to the same target assumed in Section 3.2. If different targets were used, how does this affect the interpretation of the likelihood-ranking results, and do the main conclusions persist with consistent target definitions?

  3. Predictive distribution used in the experiments. Lines 76–77 and 85–86 describe computing predictive means and variances from model outputs. How does this construction instantiate the component distributions p(y∣x,ξi)p(y\mid x,\xi_i) and their average in Eq. (5)? The released implementation computes KL and log-likelihood using a single Gaussian constructed from the selected outputs' mean and standard deviation. Please clarify whether this Gaussian is an approximation to the distribution in Eq. (5), and state the assumptions used.

3 comments

We thank the reviewer for the thorough analysis of our work. In the following we address the raised concerns.

There is little analysis of when selection improves predictive quality

Table 1 shows that KL-based sample selection with DiBNNs improves predictive quality for most BNN types, including both continual and ensemble methods. The ability to provide real uncertainty for the dataset is the most impactful dataset change to ensure the best uncertainty quality improvements. Given the results from the newly added validation set size experiments presented in Figure 2, uncertainty-labeling even a part of a validation set equivalent to 2%2\% size of the training set, we can achieve better results than by ranking on the regular validation set with the 20%20\% size of the training set.

Some motivational claims go beyond what is established

Thank you for pointing out this issue. We agree that the original formulation is too strong. We do not actually expect the approximations P(y∣x)P(y|x) and Q(y∣x)Q(y|x) to be equal, we only need the approximation P(y∣x)P(y|x) to be closer to Q(y∣x)Q(y|x) than to the approximated distribution p(y∣x)p(y|x). Since we can compute closeness between any (directed) pair of distributions, any distribution P(y∣x)P(y|x) can be treated as an approximation of any other distribution, and how well it works as an approximation depends only on the KL-divergence between them. Initially, we frame P(y∣x)P(y|x) only as an approximation of p(y∣x)p(y|x), but when we reframe it as an approximation of both p(y∣x)p(y|x) and q(y∣x)q(y|x), then we can express its closeness to q(y∣x)q(y|x) through the intermediate distribution Q(y∣x)Q(y|x) and later compare it to the true distribution q(y∣x)q(y|x) directly. This gives us a direct path from a "good approximation of the close but incorrect distribution" to a "worse approximation of the original distribution but better approximation of the true distribution. We have updated the text with the corresponding changes as follows:

For sufficiently close distributions $p(y|x)$ and $q(y|x)$, there exist samples of $p(\xi)$ and $p(\eta)$ such that KL-divergence between $P(y|x)$ and $Q(y|x)$ is smaller than between $P(y|x)$ and distribution $p(y|x)$ that it approximates, as can be seen in Figure 1.

How broadly the benefits extend beyond this setting

In our experiments we show the broad benefits and limitations of the proposed DiBNN models, but the ranking process only operates based on the samples provided by the models. For this reason, the gains are different based on the method used, and, for example, the MCD models achieve no gain and simply require the highest number of samples, which aligns with previous works analysing MCD models that show that they behave differently from other BNNs. The exact analysis of each model type and its samples will be left for future work, which is best done with a combination of synthetic and real uncertainty-labeled datasets.

Observation noise in implementation

The \textit{GPRegression} class is responsible for adding noise in the KL-based ranking scenario, and this is explicitly created for the log-likelihood ranking scenario in the \textit{find_best_samples} function because of the different evaluation routine, so both scenarios share the same targets.

Interpretation of q(y∣x)q(y|x)

The q(y∣x)q(y|x) term refers to the target distribution for each point xx. The method does not specifically assume them to be noisy, but we follow The Neural Testbed experiments and include noise in the problem.

Predictive distribution used in the experiments

We follow the common approach that is used in The Neural Testbed and estimate the Gaussian distribution based on the mean and variance of predictions of all model samples for the given input.

Thank you for the response, the revised statement in Section 3, and the new validation-size ablation. After reading the revised paper, I have lowered my Rating and Soundness to 1, because the concerns I had considered fixable remain unresolved and the new ablation raises a further one. My reasons, in order of importance, are below.

1. Label-only ("real-world") setting. The response leaves the broader analysis to future work. However, the label-only extension is listed as a contribution and presented as "Application to Real-World Datasets", while all experiments use the synthetic testbed with one sampled label per point. Section 4 also states that hyperparameters are selected by validation KL; if this applies to the LL-ranked models too, the pipeline still uses the true distribution. Together with the paper's own observation that no LL-ranked DiBNN outperforms the BNN at TT samples (Table 2), I do not think this contribution is currently supported.

2. Validation-set size (Figure 2). I appreciate the new ablation. For the smallest validation sets, validation KL is close to zero while test KL is clearly worse than at the baseline, and the LL-ranked DiVNN curve is very unstable; this suggests the greedy selection fits the validation points rather than q(y∣x)q(y|x). The figure appears to show a single training-set size, without error bars, and the scale of the right panel does not allow me to verify the statement that KL ranking with a 2% validation set outperforms LL ranking with a 20% one. A table with the numbers would help. As far as I can tell from the released code (src/experiments/load.py L87, L170), with ν=0.2\nu = 0.2 a third of the 27 configurations have only 2 validation points. Do the failures in Table 2 coincide with these configurations?

3. Observation noise. As far as I can tell from the released code, the GPRegression class adds noise only to the training labels (src/experiments/testbed.py L78–80), whereas my question concerned the evaluation targets. The KL reference appears to be the noise-free latent posterior (src/ntk/predict.py L88–89), while the LL validation labels include observation noise (testbed.py L308–310). If so, the two settings do not share the same target, which could explain part of the gap between Tables 1 and 2. Could the authors clarify this, and confirm that the released code produced the reported tables?

4. Theoretical statement. Thank you for the rewording. The revised statement is still an existence claim and relates P(y∣x)P(y|x) to Q(y∣x)Q(y|x) rather than to q(y∣x)q(y|x); since KL has no triangle inequality, closeness to QQ does not imply closeness to qq. The related claim in Section 3.1 (the full set ΞT\Xi_T "should be a worse approximation" for sufficiently high TT) and "We formally derive ..." are unchanged. I would suggest presenting these as intuition rather than derivation.

5. Minor. I accept the explanation regarding the moment-matched Gaussian; please state it in the paper. The relationship to Layer Ensembles and Caruana et al. (2004) was not addressed.

Thank you for your additional questions.

  1. We changed the section title to "Application to real-world settings". This section is intended to demonstrate how the method should be adapted to settings that are present in the usual real-world datasets, and the corresponding experiments show what the difference in expected gain is between uncertainty-labeled and normal datasets.

  2. We have clarified that we used a training set size of 100100, so the smallest validation set size is just 22 samples, which shows how the methods perform under extreme lack of data. The ranking process has access only to the validation samples, so the fewer samples we have, the harder it is to understand the true distribution, which is in line with general ML training problems.

  3. The noise (adjustable by a dataset parameter) is applied to all samples from distributions. In the KL scenario, it means that training data points have noise added to them, and in the LL scenario, it means that both training and validation data points have noise added to them since now they both are simply samples from the distribution.

  4. When TpT_p -> inf⁡\inf, P(y∣x)P(y|x) -> p(y∣x)p(y|x), and the same applies to Q(y∣x)Q(y|x) and q(y∣x)q(y|x). This means that for a given distance threshold ϵ\epsilon, there exists TpϵT^\epsilon_p such that KL(P(y∣x)∣∣p(y∣x))KL(P(y|x)||p(y|x)) < ϵ\epsilon, and the same exists for Q(y∣x)Q(y|x) and q(y∣x)q(y|x). Let us set ϵ<<KL(p(y∣x)∣∣q(y∣x))\epsilon << KL(p(y|x)||q(y|x)). We now have the following:

  1. Tp>TpϵT_p > T^\epsilon_p and Tq>TqϵT_q > T^\epsilon_q. Then, KL(P(y∣x)∣∣p(y∣x))+KL(Q(y∣x)∣∣q(y∣x))<<KL(p(y∣x)∣∣q(y∣x))KL(P(y|x)||p(y|x)) + KL(Q(y|x)||q(y|x)) << KL(p(y|x)||q(y|x)), which means that the approximations are closer to the corresponding distributions than to each other.

  2. P1(y∣∣x)P_1(y||x) is defined as a poor approximation of p(y∣x)p(y|x) by using Tp1<<TpϵT_{p1} << T^\epsilon_p, but the approximation samples are chosen in such a way that KL(P1(y∣x)∣∣Q(y∣x)))<KL(P1(y∣x)∣∣p(y∣x))KL(P_1(y|x)||Q(y|x))) < KL(P_1(y|x)||p(y|x)) and Tq>TqϵT_q > T^\epsilon_q. Then, given a small enough epsilon, KL(P1(y∣x)∣∣Q(y∣x))≈KL(P1(y∣x)∣∣q(y∣x))KL(P_1(y|x)||Q(y|x)) \approx KL(P_1(y|x)||q(y|x)). And then KL(P1(y∣x)∣∣q(y∣x)))<KL(P1(y∣x)∣∣p(y∣x))KL(P_1(y|x)||q(y|x))) < KL(P_1(y|x)||p(y|x)) if the initial difference was not tight. This part serves only as an initial step to formalization, and the actual procedure does not need the best sample set to be lower than TT. The formal derivation of the procedure works even if the total samples are the best set, and since we cannot work with infinitely high values of TT, we cannot choose ϵ\epsilon values to be as small as we want. The experiments show that the best samples are indeed much lower than even the T=1000T=1000 sample set for the tested methods, but MCD shows that it does not really have the same attributes as other BNNs, so the effectiveness of the ranking process depends on how exactly the models behave and how rich the sample sets we can work with is.

  1. We added the explanation of the predictive distribution generation to the paper. The method is based on the empirically tested approach in Layer Ensembles and extends it to all BNNs with formal derivation of the whole process for both uncertainty-labeled and regular datasets. The experiments show the comparison with the optimizations presented in Caruana et al. and how effective they are when we can test not only for accuracy but also for the quality of the estimated model uncertainty.
Reviewer_RqXvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a ranking procedure to reduce the computational burden of uncertainty estimation for Bayesian neural networks while maintaining or improving predictive performance. The main contribution is the adaptation of the existing ranking process to various BNN architectures with an extension to the case where the true predictive distribution is unavailable. The experiments on simulated data illustrate the effectiveness of the proposed method compared with standard BNNs when the true predictive distribution is available for the validation dataset.

Critical strengths
  • Clever use of ranking process : It is appealing to use greedy algorithm instead of searching all combinations in subset selection. This is reminiscent of forward selection in variable selection, which provides a practical way to estimate posterior predictive sample with limited computational budget.

  • The extension to settings where the true predictive distribution of the validation data is unavailable is good, which enhances the practicality of the proposed method.

Critical weaknesses
  • Limited novelty beyond the use of ranking process : Although the authors apply the ranking process in a proper sense, it is not sufficiently clear what distinctive methodological contribution of this work beyond the Oleksiienko & Iosifidis, 2023's approach. I think the authors should be clearer about this point.

  • Insufficient justification of the motivation : The authors explain that utilizing all the posterior samples to estimate posterior predictive distribution does not always give better result in a sense that the true predictive distribution might be different from the estimated one. This observation is reasonable, but it does not necessarily imply that a smaller subset of posterior samples provides a better approximation to the true predictive distribution. Further justification would enhance the soundness of the method.

  • Performance of log-likelihood-ranked DiBNNs : Table 2 compares the performance of DiBNNs with standard BNNs in a setting that reflects realistic scenarios where the true predictive distribution is unavailable. Comparing the DiBNN KL (first row) with the BNN KL at T samples (last row), it appears that full-sample BNNs consistently achieve lower KL divergence than DiBNNs across all six architectures. If this interpretation is correct, these results do not seem to support the claim that selecting a subset of posterior samples can potentially improve predictive performance over using the full sample set, particularly in this practical setting.

Questions for authors

Q1. In Sec 3.2, if the number of the validation sample is small, is there any risk of overfitting during the ranking procedure?

Q2. Could the authors clarify whether discrete BNNs is computationally cheaper than the standard Monte Carlo estimator when we also take into account the overall ranking procedure?

3 comments

We thank the reviewer for the thoughtful evaluation. In the following, we answer to the raised concerns and questions.

Limited novelty

We provide the following improvements over Oleksiienko & Iosifidis, 2023:

  • We formally define the sample ranking process for all types BNNs and show that it works not only on a single ensemble approach but on a wide variety of methods, including both continuous and ensemble BNNs.
  • We propose the log-likelihood-based search and connect it to the KL-based search as being a direct approximation of it. We show where this leads us to the improvement in the final models and where it falls short.
  • We evaluate different ranking optimizations and show how they behave under both the uncertainty-labeled and regular datasets.

Insufficient justification of the motivation

We base this assumption on a notion of "sufficiently high value of TT", which means that the Monte Carlo Integration process will lead to computationally exact values once TT is a high enough number, and the closer we are to this value, the smaller is the divergence between the approximation and the predictive distribution. If we find such TT that the distance between the predictive and the true distributions is higher than the distance between the approximation and the predictive distribution, then increasing TT will move us further away from the true distribution. The issue is that it is impossible to state exact limits of TT and reasonably operate with those distances for BNNs. However, we can see that if we have uncertainty labels for the validation set, the T=1000T=1000 is enough to be too close to the predictive distribution and too far from the true distribution, as only 3535-150150 samples are needed for VNNs, BBB, and Hypermodel continuous methods.

Performance of log-likelihood-ranked DiBNNs

We agree with the reviewer that log-likelihood-based ranking does not provide the same improvement as KL-based ranking, but even then, if we follow the ranking process, the resulting models achieve better uncertainty quality at the selected number of samples than the corresponding BNNs. This means that if we have unlimited computational resources, we should just use the highest number of samples for BNNs, but in most cases we are highly limited in the number of samples we can apply during inference. Practically, BNNs will not be able to run more than 1010 samples for one input for latency-constrained tasks such as robotics navigation. If we are forced to use fewer samples, then DiBNNs can provide better uncertainty than BNNs.

Q1. Validation set size influence

We thank the reviewer for this important question. We implemented ablation experiments with Deep Ensembles and VNN models. We added Figure 2 to the paper, which shows the effect of the validation size on DiBNN performance for both ranking processes. The experiments show that when the validation set size is smaller than 2%2\% of the training data, the KL-based models achieve around two times worse quality of uncertainty compared to the 20%20\% baseline, and the 100%100\% models are slightly better than the baseline.

Q2. Computational complexity

The ranking is done once after training and is reused for all future inferences. Ranking requires T×∣Dv∣T \times |D_v| model inferences and the greedy search based on the stored model outputs. Since we can use the ranked models to achieve better uncertainty at lower samples, we expect to use fewer computations for KL-ranked models and expect to have better uncertainty for the same limited computations for log-likelihood-ranked models.

Firstly, I would like to thank the authors for the detailed responses to my concerns and questions.

(1) Limited novelty:

I still consider the methodological novelty of this work to be relatively limited, while acknowledging its practical contributions. My original concern has been partially addressed.

  • Considering the fact that the iterative selection procedure in Oleksiienko & Iosifidis (2023) is formulated in a general manner, it is somewhat unclear whether extending it to other BNN architectures requires any substantial methodological modification.

  • The log-likelihood based search provides a pragmatic scheme when true predictive distribution is unknown. However, the argument that leads to log-likelihood maximization follows from a standard identity.

(2) Insufficient justification of the motivation:

I understand and agree with the rationale behind the proposed methodology, particularly the possibility that the predictive distribution differs from the true predictive distribution.

After reading the response, I wonder whether the effectiveness of the proposed method depends on the degree of discrepancy between the BNN predictive distribution and the true predictive distribution. In particular, I suspect that the proposed method may be more beneficial when this discrepancy is larger. A brief discussion of this point would help clarify the circumstances under which the proposed method is most effective.

However, if the authors do not share this intuition, I do not consider the addition necessary. My original concern has largely been resolved by the authors' explanation.

(3) Performance of log-likelihood-ranked DiBNNs:

My concern regarding this point has been resolved. I agree that the proposed method can be beneficial in computationally constrained settings, where only a limited number of samples can be used during inference. The results in Table 2 support this interpretation when comparing DiBNNs with standard BNNs using the same number of samples.

(4) Validation set size influence:

Thank you for conducting the additional ablation experiments. I believe Figure 2 provides useful evidence regarding the sensitivity of the proposed method to the validation set size and thus offers practical guidance on the choice of validation set size. This adequately addresses my question.

(5) Computational complexity:

I understand that the ranking procedure is performed only once and that its cost can be amortized over subsequent inferences. I believe this adequately addresses my question, although a brief discussion of the one-time ranking cost versus the inference savings would further clarify the overall computational trade-off.

Thank you for your response.

Whether extending it to other BNN architectures requires any substantial methodological modification Layer Ensembles only establish this process for ranking of ensemble-based models that have a fixed limited set of samples, and before this work, it was unclear whether the approach may work at all for continual BNNs since they have an unlimited number of samples and their samples may not be as different as the ones in ensemble methods (due to not being tied to a single defined distribution such as a Gaussian). The application to usual BNNs requires discretization of the model and breaks the expected stochasticity of BNNs.

Log-likelihood based search Yes, the definition of it is straightforward, but it is still unclear how viable this is before the experiments we presented, with direct comparison with true distributions and the KL-based ranking, instead of using secondary metrics due to the lack of useful testing data.

Whether the effectiveness of the proposed method depends on the degree of discrepancy between the BNN predictive distribution and the true predictive distribution The discrepancy between the true and the predictive distribution affects the effectiveness of BNNs in two opposite ways: if the BNN is perfect, then ranking is useless, but if the BNN is too far from the truth, then no ranking will help it enough, so DiBNNs will most likely be a tool to make a good BNN better, not fix the original model's inability to operate.

Computational complexity We have also added a table that measures the computational complexity of a ResNet-152 classical, VNN and BBB models and the corresponding ranking costs at 2000020000 validation images.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.