- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a multi-encoder ensemble and a linear Bayesian layer head for calibrated uncertainty estimation on regression problems. The method is referred to as Bayesian Deep Kernel Networks (BDKN). Aside from this framework, the group claims BDKN meaningfully differs from Bayesian last-layer (BLL) in its uncertainty analysis over multiple feature vectors. The last contribution/claim is that weighted aggregation down weights weaker members of the ensemble, which benefits overall robustness. Evaluation is performed on selected UCI regression benchmark datasets.
The overall topic is likely to be of interest to readers working in this field (uncertainty quantification), and the manuscript is clearly written and easily digestible. The novelty is moderate, and the results while testing a number of methods, are not fully convincing of its effectiveness in practical settings.
Critical strengths
- The paper is well written, and easy to follow.
- The paper focuses on an interesting and important domain pertaining to uncertainty quantification, and seeks to develop a viable means of capturing epistemic uncertainty in the encoder step prior to BLL.
Critical weaknesses
-
On novelty, the work is not a significant development but instead incorporates a number of existing ideas. The work is ensembled version of BLL.
-
One interesting idea is the idea of a learned weighted aggregation. However, this doesn't seem to be fully explored:
i) If it's just one or two networks dominating, and the weaker networks are made irrelevant by small weights, why not simply choose the best network, drop the rest, and avoid aggregation altogether?
ii) A7 compares the weighting to unweighted DE. What happens if you weight the DE?
iii) On this point, is this really a fair comparison to BLL? It seems to be effectively increasing the number of chances of getting a strong encoder for a dataset then applying BLL.
-
On results, there is a lack of significant results and a lack of thorough statistical testing, showing standard deviation alone. On a number of these, results do not seem . But more importantly there are no paired tests, which might actually aid the author in this case, in discerning significance.
-
Performance in regression is not necessarily the main area of importance for uncertainty measures. There exists no primary calibration metrics reported, making claims about calibration unreliable. On NLL (L252):
i) NLL is not a dedicated calibration measure, and does not sufficiently reflect calibration on its own.
ii) NLL is also difficult to interpret since it depends on the dataset.
-
It is indeed the case that UCI datasets are often used as the benchmarks in these types of works, but performance and calibration on curated datasets which are small in both sample size and dimensionality do not necessarily translate well to real-world results.
i) As noted in [R1], a significant area of interest for uncertainty application is on OOD data. The current work does not demonstrate any capacity in this setting.
Additional Comments
- GP with RBF kernel is probably not viable on datasets like Song even if it could fit on GPU, because of the curse of dimensionality.
- Related to W5, it is not clear how effective this approach would be for modalities where uncertainty is of high interest, such as vision, language, or sound, where raw data is used for important diagnostic and identification work. Consider the case where a shortcut is exploited by all ensemble members due to the nature of the training data. Under a distribution shift, the entire ensemble fails and is confidently incorrect. Which is an issue DUMs attempt to solve [R2].
- I was not able to find a declaration of AI usage.
Minor issues
- The last line of extending the work to RL and PINNs is not valuable to the paper.
- While it is understandable why the authors chose Jeffrey's prior, the claim that it improves generalizability and uncertainty quantification (L177) is vague and may be overstating things.
- The cost and "scalability" of DE is somewhat understated and overstated respectively. This comes down to the datasets and modalities being used.
Recommendations
- Clarifications for W2.
- Paired tests to establish significance (W3).
- One or two tests on real world uncurated datasets (do not have to be high compute or in full, W5), learned features, and with proper calibration evaluation (combining W4) would strengthen this work.
- Adding calibration measures for all tested methods and datasets (W4).
Questions for authors
My questions have been combined with the weaknesses above.