acceptodds
Withdrawn from ICLR 2027

Scalable Bayesian Deep Kernel Networks via Ensemble-Induced Finite Function Spaces

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce Bayesian Deep Kernel Networks, a framework for predictive regression designed to enhance interpretability while maintaining competitive predictive performance and computational efficiency. Our method combines the statistical rigor of Bayesian inference with the scalability of deep ensembles, providing calibrated uncertainty estimates that enable its use not only for standalone prediction but also as a component within broader learning systems. To achieve these goals our work relies on three key design components: (i) low-dimensional ensemble representation: predictions are expressed as a combination of a small number of trained neural predictors, enabling scalable inference whose cost depends on ensemble size rather than dataset size; (ii) closed-form Bayesian aggregation: ensemble predictions are combined using Bayesian linear regression, yielding interpretable posterior weights and calibrated uncertainty without approximate inference; and (iii) Independent ensemble training: multiple neural networks are trained separately, producing diverse predictive representations that improve robustness and uncertainty calibration. Empirical results on standard regression benchmarks demonstrate that the proposed approach achieves competitive predictive performance while maintaining reliable uncertainty estimates across settings.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian neural network, uncertainty quantification, functional modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hrUxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a multi-encoder ensemble and a linear Bayesian layer head for calibrated uncertainty estimation on regression problems. The method is referred to as Bayesian Deep Kernel Networks (BDKN). Aside from this framework, the group claims BDKN meaningfully differs from Bayesian last-layer (BLL) in its uncertainty analysis over multiple feature vectors. The last contribution/claim is that weighted aggregation down weights weaker members of the ensemble, which benefits overall robustness. Evaluation is performed on selected UCI regression benchmark datasets.

The overall topic is likely to be of interest to readers working in this field (uncertainty quantification), and the manuscript is clearly written and easily digestible. The novelty is moderate, and the results while testing a number of methods, are not fully convincing of its effectiveness in practical settings.

Critical strengths
  1. The paper is well written, and easy to follow.
  2. The paper focuses on an interesting and important domain pertaining to uncertainty quantification, and seeks to develop a viable means of capturing epistemic uncertainty in the encoder step prior to BLL.
Critical weaknesses
  1. On novelty, the work is not a significant development but instead incorporates a number of existing ideas. The work is ensembled version of BLL.

  2. One interesting idea is the idea of a learned weighted aggregation. However, this doesn't seem to be fully explored:

    i) If it's just one or two networks dominating, and the weaker networks are made irrelevant by small weights, why not simply choose the best network, drop the rest, and avoid aggregation altogether?

    ii) A7 compares the weighting to unweighted DE. What happens if you weight the DE?

    iii) On this point, is this really a fair comparison to BLL? It seems to be effectively increasing the number of chances of getting a strong encoder for a dataset then applying BLL.

  3. On results, there is a lack of significant results and a lack of thorough statistical testing, showing standard deviation alone. On a number of these, results do not seem ≥2σ\ge 2 \sigma. But more importantly there are no paired tests, which might actually aid the author in this case, in discerning significance.

  4. Performance in regression is not necessarily the main area of importance for uncertainty measures. There exists no primary calibration metrics reported, making claims about calibration unreliable. On NLL (L252):

    i) NLL is not a dedicated calibration measure, and does not sufficiently reflect calibration on its own.

    ii) NLL is also difficult to interpret since it depends on the dataset.

  5. It is indeed the case that UCI datasets are often used as the benchmarks in these types of works, but performance and calibration on curated datasets which are small in both sample size and dimensionality do not necessarily translate well to real-world results.

    i) As noted in [R1], a significant area of interest for uncertainty application is on OOD data. The current work does not demonstrate any capacity in this setting.

Additional Comments

  1. GP with RBF kernel is probably not viable on datasets like Song even if it could fit on GPU, because of the curse of dimensionality.
  2. Related to W5, it is not clear how effective this approach would be for modalities where uncertainty is of high interest, such as vision, language, or sound, where raw data is used for important diagnostic and identification work. Consider the case where a shortcut is exploited by all ensemble members due to the nature of the training data. Under a distribution shift, the entire ensemble fails and is confidently incorrect. Which is an issue DUMs attempt to solve [R2].
  3. I was not able to find a declaration of AI usage.

Minor issues

  1. The last line of extending the work to RL and PINNs is not valuable to the paper.
  2. While it is understandable why the authors chose Jeffrey's prior, the claim that it improves generalizability and uncertainty quantification (L177) is vague and may be overstating things.
  3. The cost and "scalability" of DE is somewhat understated and overstated respectively. This comes down to the datasets and modalities being used.

Recommendations

  1. Clarifications for W2.
  2. Paired tests to establish significance (W3).
  3. One or two tests on real world uncurated datasets (do not have to be high compute or in full, W5), learned features, and with proper calibration evaluation (combining W4) would strengthen this work.
  4. Adding calibration measures for all tested methods and datasets (W4).
Questions for authors

My questions have been combined with the weaknesses above.

Reviewer_GVF8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors propose a novel method for training probabilistic regression models that retain both the effectiveness of deep ensembles and the grounding of bayesian regression approaches.
In a nutshell, the proposed method attempts to train models in a deep ensemble fashion as feature maps that are subsequently used for Bayesian aggregation.
The proposed method showed promising results on a number of tabular regression benchmarks, outperforming other approaches in terms of prediction accuracy and the quality of uncertainty estimates.

Critical strengths
  • The paper is clearly written, the method is logical, and the authors properly address the motivation behind introducing a new type of probabilistic model based on ensembles.
  • The paper tackles an important and timely problem of uncertainty quantification in deep learning models, which has critical use cases in real-world applications.
  • Compared to ensembles, the proposed approach is efficient, requiring a lightweight aggregation of ensemble model predictions.
Critical weaknesses
  • The proposed approach is specifically designed and benchmarked only with regression in mind, though it could be extended to classification, which significantly limits its scope.
  • While ensembles indeed could be considered a gold standard in terms of uncertainty quality and model accuracy, as was also mentioned in the paper, they are pretty expensive and therefore can’t be used in many real-world applications. A large body of literature tackles this issue by introducing lightweight ensembling approaches, such as [1, 2, 3], which are not discussed in the paper.
  • The evaluation mostly considers small-scale regression datasets; therefore, evidence that the proposed approach indeed scales to larger architectures and tasks is limited.
  • Aggregating ensemble predictions is not entirely new, and prior work has explored this direction [4]. A clearer discussion of how the proposed aggregation differs from these approaches would help establish its novelty.

[1] Wen, Yeming, Dustin Tran, and Jimmy Ba. "Batchensemble: an alternative approach to efficient ensemble and lifelong learning." arXiv preprint arXiv:2002.06715 (2020).

[2] Durasov, Nikita, et al. "Masksembles for uncertainty estimation." 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021.

[3] Laurent, Olivier, et al. "Packed-ensembles for efficient uncertainty estimation." arXiv preprint arXiv:2210.09184 (2022).

[4] Liu, Jeremiah, et al. "Accurate uncertainty estimation and decomposition in ensemble learning." Advances in neural information processing systems 32 (2019).

Questions for authors
  • Could you evaluate the proposed approach on larger datasets and more complex tasks? ImageNet classification could serve as a standard benchmark, while a dense prediction task such as semantic segmentation would provide further evidence of its broader applicability.
  • What modifications would be needed to extend the method to classification?
  • How does the method compare with lightweight ensembling approaches [1, 2, 3] in terms of predictive accuracy, uncertainty quality, training cost, and inference cost? Can proposed approach be combined with these approaches?
  • How does the proposed Bayesian aggregation differ from similar prior work on aggregating ensemble predictions, e.g. [4]?
Reviewer_QanCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces Bayesian Deep Kernel Networks (BDKN) for regression. Standard deep ensemble (DE) trains several neural networks independently and averages their predictions with equal weights. BDKN instead use each network’s predicted mean as a basis function in a Bayesian linear regression. By this, the model can also learn how to weight the individual predictors. Ideally, the method can reduce the influence of a poorly fitted network while retaining predictors whose errors offset one another. With fixed networks, BDKN integrates over the aggregation weights and observation-noise variance analytically. This yields a Student-(t) predictive distribution that accounts for uncertainty about how to combine the networks. The additional computation is modest for small ensembles because the linear system to be solved has one unknown per network.

The paper evaluates BDKN on 11 regression benchmarks and on additional splits that withhold regions of the input space from training. The largest gains over equal-weight deep ensembles occur when some networks fit the training data poorly and produce extreme predictions. BDKN reduces their influence at little additional computational cost. However, under a more conservative training schedule, the differences in prediction accuracy are generally smaller, and the standard deep ensemble achieves lower mean negative log-likelihood on most benchmarks.

Critical strengths
  • The main question the paper in the paper is practical, asking how should we combine predictors when their quality varies. The proposed solution is simple, and ideally should work well under simple settings. The method can be applied to an already trained ensemble, and the Bayesian aggregation step yields a closed-form predictive distribution at little additional computational cost. These support the readability and soundness of the paper.
  • Some good experimental results make the comparison easy to interpret. By applying both aggregation rules to the same trained networks, the authors isolate the effect of how the predictions are combined. Evaluating both training regimes helps identify the conditions under which learned weighting is most beneficial. This supports the soundness of the paper.
Critical weaknesses
  • Although some of the experimental results are good, the experiments are overall quite weak to support the gain of the method over classical results. The strongest gains occur under an aggressive training regime in which some ensemble members fit poorly. Under the more conservative regime, BDKN has worse reported mean NLL than the standard deep ensemble on eight of the eleven regression datasets (Table 10) and all eight gap benchmarks (Table 14).
  • Benchmarks could be made richer, so that the experimental results could be more convincing. The main ensemble comparison uses equal weights, while BDKN learns its aggregation coefficients from data. I therefore recommend adding other benchmarks that also learn the weights for predictors (such as Stacked Regressions (Breiman, 1996), or Stacking of predictive distributions (Yao et al., 2018)).
Questions for authors
  • How do the authors interpret BDKN’s worse mean NLL than DE on eight of eleven datasets under conservative training (Table 10) and all eight gap benchmarks (Table 14)?
  • How does BDKN compare with methods that also learn aggregation weights, for example stacked regression (Breiman, 1996) or stacking of predictive distributions (Yao et al., 2018), when applied to the same ensemble?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.