acceptodds
Under review as a conference paper at ICLR 2027

ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty

Abstract

Claims about predictive uncertainty are hard to check against real data: each example typically carries one realized label, never the distribution behind it, so uncertainty quantification is often judged against estimates rather than truth. We introduce ProbabilityBench with three simulated worlds, i.e., credit review, peer review, and covert transfer, where partial observability creates irreducible uncertainty, natural-language rendering provably preserves it, and every case's exact class probability is known. Such ground truth has existed only in toy settings such as Gaussian class conditionals, unsuited to testing a language model. With ProbabilityBench, a model's probability is scored by its exact KL divergence, e.g., Jev is both the cheapest and the closest to the true probability in every world among the models we compared. Ground-truth probabilities also enable us to evaluate the evaluation methods themselves, to assess how accurate uncertainty-quantification methods such as calibration estimators and Bayes-error estimators actually are.

Primary area:
datasets and benchmarks
Keywords:
model evaluation, calibration, irreducible error, Bayes error

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_48K7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce ProbabilityBench, a collection of synthetic uncertainty benchmarks for language models with the property that all quantities of interest---including exact class probabilities---are known exactly. Evaluations span several domains, including finance and AI safety. They run a series of evaluations, concluding that recent "decision"-style models like Jev are Pareto-optimal.

Critical strengths

It's refreshing on some level to see an uncertainty benchmark that involves more than just increasing accuracy on a factual recall task; quantities like the ones exposed here are precisely what uncertainty estimates are supposed to approximate, and so we should be measuring them directly. The paper is fairly straightforward and easy to follow.

Critical weaknesses

Many uncertainty benchmarks, as I've already mentioned, have the flaw that they focus on epistemic uncertainty and neglect aleatoric uncertainty completely---on TriviaQA, for example, there is essentially only one correct answer, recognizable from the very first token. This paper practically has the exact opposite problem. Models are judged here according to the divergence between their predictions and the true conditional distribution, but this metric conflates genuine failures to properly represent aleatoric uncertainty with epistemic failures to learn the underlying (often quite complex) decision rule. As far as I can tell, the paper does very little to tease these two failure modes apart. The problem is exacerbated by the fact that models are only exposed to about 50 examples per rule. Several quantities would be useful here: the optimal prediction achievable by any learner with access to just those 50 examples, experiments with a higher number of samples, results of experiments with models that are directly told the decision rule, etc.

The main figures (e.g. 2) are missing elementary baselines (constant, logistic regression, etc.).

The paper tries to distinguish itself from toy settings, but I'm not sure if it clears that particular bar. All three "worlds" are just lightweight natural-language wrappers around synthetic decision rules, and it's not clear to me that results we see here will transfer very cleanly to much messier language tasks. It may be a step in the right direction, but this is still a toy task. A large chunk of the experiments (Figures 6, 7, and 8) don't use LLMs at all.

Questions for authors

See above.

Reviewer_KzNuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ProbabilityBench, a benchmark consisting of three simulated worlds with exactly computable conditional label probabilities and Bayes errors. The three simulated worlds are: financial default prediction, which weighs a company’s resources against its obligations; peer-review outcome prediction, which scores a manuscript’s quality against an acceptance bar; and an AI-safety audit, which asks whether an AI system hid a confidential tag in a file name others can see. The authors use the benchmark to evaluate model confidence, Bayes-error estimators, and calibration methods, demonstrating discrepancies between estimated uncertainty and the known ground truth.

Critical strengths
  1. Unlike benchmarks that rely on proxy measures or estimated uncertainty, ProbabilityBench provides exact conditional probabilities and Bayes errors. This enables direct evaluation of probability predictions and uncertainty estimators.

  2. The benchmark supports analyses of Bayes-error estimation, multicalibration, and higher-order calibration. The distinction between realized and expected accuracy is particularly useful for examining test-set overfitting.

Critical weaknesses

Although the benchmark uses natural-language tasks, the underlying worlds are manually specified finite-state systems, and the text is generated from deterministic templates. The resulting tasks may primarily test whether models can infer relatively simple hidden rules from structured descriptions, rather than uncertainty estimation in realistic language understanding or decision-making. The exact probabilities are a clear advantage, but it remains unclear whether the observed model rankings and estimator failures generalize beyond these synthetic settings. The benchmark is capable of evaluating several uncertainty-related quantities, but many reported findings are already expected: nearest-neighbor Bayes-error estimators struggle in sparse high-dimensional spaces, finite-sample calibration estimates are noisy, and test-label access enables overfitting. The paper would be more compelling if it demonstrated a failure mode or methodological conclusion that existing evaluation protocols would systematically miss, rather than mainly confirming known limitations under controlled conditions.

Questions for authors
  1. How sensitive are the model rankings and uncertainty evaluation results to the manually chosen generative rules, factor distributions, and rendering templates?

  2. What conclusions from ProbabilityBench are expected to transfer to real-world uncertainty estimation, beyond the ability to evaluate against exact probabilities?

  3. Can the authors demonstrate a case where conventional calibration metrics suggest that a model is reliable, while the exact conditional probabilities reveal a substantially different conclusion?

Reviewer_f88oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work introduces a benchmark to evaluate the accuracy of uncertainties evaluated by language models which they call ProbabilityBench. The work is specifically targeted to real-world situations. In their setup, to bypass the fact that in realistic scenarios one can only access the one ground true label that was realized as opposed to the true conditional (posterior) distribution P(y∣Xvis)P(y \vert X_{vis}), they restrict the benchmark to cases where the true Bayesian posteriors can be calculated analytically. In practice, this is done by limiting the inference to one-dimensional scenarios where the Y variable is binned and binary, making the evaluation tractable analytically over a 1-d, binary, discrete distribution. The authors then compute the KL-divergence between this analytical binned distribution and the one outputted by the tested models. This framework is exemplified in three synthetic worlds, Financial Default, Peer-Review Outcome, and Covert Information Transfer, where ground truth latent, nuisance variables, and deterministic outcome rules are fully specified.

Critical strengths
  • There is a clear and important conceptual motivation and the paper addresses a clear need. Moreover, the approach of using empirical binning and introducing a tractable target distribution that can be evaluated analytically is intuitive and appealing for evaluating the calibration of the uncertainties provided by models in text-based reasoning.
  • The mathematical construction of the three synthetic worlds is rigorous and thoroughly explained, making the calculation of class probabilities easy to reproduce.
  • The paper demonstrates empirically a possible failure mode of non-parametric estimators by showing how standard Bayes error estimators fail to extract all available information and provides under confident error estimates when applied to high-dimensional text-derived observation spaces contaminated by nuisance parameters.
  • The paper introduce a metric that can detect test set overfitting by quantifying the expected versus realized accuracy gap, and they demonstrate that it can detect overfitting when models are given access to test labels.
  • The paper includes cutting-edge frontier models in their benchmark, which gives immediate empirical relevance to the work. However, this is also a double-edge sword since frontier models are a moving target that is constantly updated, meaning it might quickly become difficult to reproduce the results presented in the paper (e.g. Fig 2).
Critical weaknesses
  • Omission of modern coverage tests and overstated claims: This is particularly apparent the way the authors overstate what is actually contributed by the paper, saying that assessing "Whether a model’s probabilities are right, and how much of its error is irreducible, can only be settled where the true class probabilities are known, which has usually meant simple synthetic distributions such as Gaussian distributions. ". This ignores established literature in the field of inference and statistics, especially the vast body of literature in Simulation Based Inference (SBI) and quantitative probabilistic modelling where coverage tests (e.g. Expected Coverage Probability test like TARP and High Density Region validation) are routinely used to rigorously evaluate the accuracy of predictive estimators. In addition to the probabilities from the model to be tested, these only require real data and ground truth pairs, with no analytical knowledge of the underlying data-generating probabilities. Given this and the fact that the models output probabilistic predictions per class, this complete absence is a huge missed opportunity given the claims are to establish a comprehensive uncertainty quantification benchmark.

  • The authors spend a good amount of real paper real estate demonstrating that standard non-parametric Bayes error estimators (like kNN, 1NN, etc) severely overestimate the true Bayes error, especially when including nuisance parameters (the irrelevant fields) in the observation vector. It is very surprising that the paper treats this as either a new finding or a purely empirical observation as opposed to explaining why this happens, as it is completely expected behaviour. Indeed, non-parametric Bayes error estimators operate on the exact same principle as Approximate Bayesian Computation (ABC), by assuming that proximity in data space X reflects uncertainty in the underlying parameter/label space Y. In this setup, considering high-dimensional feature spaces with irrelevant or nuisance dimensions means the dimensions containing information about the parameters are diluted (especially if considering distance-based metrics like Euclidean kNN) and will create wide, uninformative ‘acceptance bands’. This is a behaviour that is both explained and expected when using non-sufficient summary statistics of the data. The fact that the paper completely ignores this theoretical framing instead of interpreting it through the lens of inference in terms of summary statistics and ABC acceptance tolerance makes it read as shallow and not complete work, especially considering the publication venue.

  • The paper presents 3 separate worlds to make their benchmark, however, it’s hard to understand the specific choices of inference problems given there is an exact isomorphic structure of the mathematical problem across all three worlds. Indeed, despite being dressed in distinct natural language domain stories in finance, peer-review, and covert information transfer, all three problems share the exact same mathematical structure. This means that, at the hearth of it, they are 3 repetitions of the same problem. They are all 1-dimensional discrete binary inference problems under partial observability, corrupted by independent noise or residual nuisance parameters. The benchmark does not present inference problems designed to test models under different inference problems, like complex multi-dimensional posteriors that could have interesting correlation structures, non-linear decision boundaries, or hierarchical inference tasks (e.g. high-dimensional observations mapped to a single sample from a potentially high-dimensional parametrized distribution, with the task of inferring hyper-parameters from that later underlying distribution). This limitation restricts the paper to evaluating 1d discrete cases, and limit its ability to be a useful benchmark for realistic tasks that often present structurally rich, diverse probabilistic reasoning.

  • After reading the paper, I am left wondering what insights (if anything) of the problems with synthetic irrelevant nuisance parameter and artificial setups would transfer to unconstrained natural language domains.

Questions for authors
  • Can you provide coverage tests and include discussion of their relevance to accompany accuracy assessment?
  • Can you reframe the discussion and findings in more standard statistical inference framework to bridge the domain gap between statistical inference expertise and natural language as opposed to increasing it?
  • The paper would be strengthen if more diverse inference tasks with varied mathematical structures were presented in the benchmark. In its current form, the work feels too preliminary to warrant publication at ICLR

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.