- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors introduce ProbabilityBench, a collection of synthetic uncertainty benchmarks for language models with the property that all quantities of interest---including exact class probabilities---are known exactly. Evaluations span several domains, including finance and AI safety. They run a series of evaluations, concluding that recent "decision"-style models like Jev are Pareto-optimal.
Critical strengths
It's refreshing on some level to see an uncertainty benchmark that involves more than just increasing accuracy on a factual recall task; quantities like the ones exposed here are precisely what uncertainty estimates are supposed to approximate, and so we should be measuring them directly. The paper is fairly straightforward and easy to follow.
Critical weaknesses
Many uncertainty benchmarks, as I've already mentioned, have the flaw that they focus on epistemic uncertainty and neglect aleatoric uncertainty completely---on TriviaQA, for example, there is essentially only one correct answer, recognizable from the very first token. This paper practically has the exact opposite problem. Models are judged here according to the divergence between their predictions and the true conditional distribution, but this metric conflates genuine failures to properly represent aleatoric uncertainty with epistemic failures to learn the underlying (often quite complex) decision rule. As far as I can tell, the paper does very little to tease these two failure modes apart. The problem is exacerbated by the fact that models are only exposed to about 50 examples per rule. Several quantities would be useful here: the optimal prediction achievable by any learner with access to just those 50 examples, experiments with a higher number of samples, results of experiments with models that are directly told the decision rule, etc.
The main figures (e.g. 2) are missing elementary baselines (constant, logistic regression, etc.).
The paper tries to distinguish itself from toy settings, but I'm not sure if it clears that particular bar. All three "worlds" are just lightweight natural-language wrappers around synthetic decision rules, and it's not clear to me that results we see here will transfer very cleanly to much messier language tasks. It may be a step in the right direction, but this is still a toy task. A large chunk of the experiments (Figures 6, 7, and 8) don't use LLMs at all.
Questions for authors
See above.