- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper asks whether the standard metrics used to benchmark uncertainty quantification (NLL, Brier score, accuracy, ECE/MCE, retention-curve AUC, error detection) rank models by their usefulness for downstream decisions. It defines decision-alignment: a metric is aligned with a parametric family of utilities if it is a strictly increasing transform of the expected utility under some prior. This is shown to be equivalent to preserving the order and ties of that expected utility.
The authors audit 32 metric–utility pairs: binary cost-sensitive decisions, top-k selection, selective prediction, one-vs-rest multiclass decisions and multivariate selective prediction. They conclude that every metric is either not aligned or aligned only under an "implausible" implicit prior.
As a remedy they propose prior-weighted utility (PWU) metrics. The expected utility under an explicitly chosen (Beta) prior. These are aligned and proper by construction. Experiments on 40 datasets report that PWUs give the highest rank agreement with downstream utility and the lowest regret when the top-ranked model is deployed.
Critical strengths
Paper turns integral representation of a proper scoring rule into a general criterion. Metric alignment as a property of a triple of metric, utility function and a prior on utility parameters.
This work also applies this principle in a large case study, for non-separable utilities, top-k selection, selective prediction in regression, and multi class and multivariate tasks.
Experimental section is also quite substantial, conducting a lot of experiments, highlighting the strength of proposed method.
Critical weaknesses
To the best of my understanding the tackled problem has been already largely studied in the literature. The binary-decision metric PWU‑BD coincides, up to the constant with the Beta-weighted proper scoring rule of Buja et al. . Reading NLL, Brier score and accuracy as expected cost under implicit cost distributions is established (Schervish 1989; Hernández‑Orallo et al. 2012).
Buja et al - Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications, https://www.semanticscholar.org/paper/Loss-Functions-for-Binary-Class-Probability-and-and-Buja-Stuetzle/d6706b6e626c15680688b0774419662f2341caee
Schrevish 1989 - A General Method for Comparing Probability Assessors - https://projecteuclid.org/journals/annals-of-statistics/volume-17/issue-4/A-General-Method-for-Comparing-Probability-Assessors/10.1214/aos/1176347398.full
Jos´e Hern´andez-Orallo et al - A Unified View of Performance Metrics: Translating Threshold Choice into Expected Classification Loss - https://www.jmlr.org/papers/volume13/hernandez-orallo12a/hernandez-orallo12a.pdf
Without this contribution, in my opinion, theoretical results stay limited.
Questions for authors
My main concern is with novelty of the proposition, it looks like the question has been already looked at in the literature. Could you help me position your work better in what have already been done. Maybe you could highlight the part of your work that can clarify the positioning exactly.
Apart from this I appreciate the paper’s goal of making UQ evaluation decision-aware, and I think the framework is a useful lens. I would like to understand better how a practitioner or benchmark designer should use it in the general-purpose setting the paper targets, where no specific downstream decision is known.
For general-purpose benchmarking, where no downstream decision is known, which decision families and priors would you recommend as a default PWU panel, and why?
If PWUs for different decision families rank methods differently, how should a benchmark summarise or interpret the results?
With no deployment context, a uninformative prior ( i.e. uniform prior ) reduces PWU‑BD to the Brier score. What do PWUs add in this case, and when is an asymmetric default such as Beta(2,10) justified?
Can the framework evaluate uncertainty scores, such as epistemic vs. total uncertainty or mutual information, and not only full predictive distributions?
In general evaluation one would not have any information about the downstream decision. You show experimntally that even in that case your PWU metric is aligned with decision utility, could a theoretical result of that flavor be obtained ? For example showing that PWU is better aligned with some unknown PWU instantiation than classical metrics ?
Rebuttal Summary
Dear reviewers,
thank you for your assessment of our work and engagement in the rebuttal. In response to your feedback, we made the following main changes to our submission: We
We appreciate your helpful feedback and your time for reviewing our paper.