- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
LLM leaderboards usually report point estimates without uncertainty, and this paper responds with Amortized Bayesian Reliability Estimation. A model's per-item benchmark scores are treated as i.i.d. draws given a latent performance parameter θ, Bernoulli for pass/fail scoring and a mean-parameterized Beta for continuous scores, with a Beta prior on θ. A permutation-invariant network trained by neural posterior estimation on simulated or historical evaluation tasks maps a set of scores to a Beta approximation of the posterior in one forward pass. The posterior mean is reported with credible intervals, and models are compared through the probability of superiority, a ROPE probability and rank distributions; a hierarchical variant pools across benchmark subgroups. The claimed contributions are this Bayesian metric, reliability-aware model comparison, and an empirical evaluation on PolyGemma, BERT and Polyglot. The abstract and Section 2.6 describe a different formulation, with a Gaussian prior over a latent semantic-quality variable and an LLM judging semantic relatedness. On validation, Sections 4.12–4.14 set out a protocol, ablations and an experimental setup, but no experimental results are reported.
Critical strengths
Uncertainty in LLM evaluation is a real problem worth solving. Models are routinely ranked on point estimates from benchmarks of a few hundred items, and a practical way to report how far those numbers can be trusted would improve how comparisons are made. The reporting design in Section 4 is sensible: it keeps the estimate separate from its uncertainty, plans to check calibration with coverage curves at several nominal levels, and fixes the ROPE margin before the data are seen. This carries limited weight in my assessment, because none of it is validated in the paper and these reporting practices already exist in the literature on Bayesian evaluation.
Critical weaknesses
The paper does not report any experiments. There are no result tables or figures, and the experimental setup in Section 4.14 still contains notes such as "[Add exact model variants/sizes and checkpoints ...]" and a request to list the benchmark datasets. It looks as though the experiments were never fixed, let alone run. Even so, the abstract says the metric was evaluated across several models and benchmarks and that "our experimental analysis demonstrates" uncertainty that point estimates hide, and a comprehensive empirical evaluation is listed as the third contribution. Other parts of the paper are unfinished as well: Section 1.1 is the ICLR template text, the reference list is empty, and Section 2 refers to the works it discusses only as "Paper 1" to "Paper 6". This on its own decides my recommendation, because none of the paper's claims can be checked.
Setting the missing experiments aside, I could not work out what the method actually is, and the version described in most detail has a flaw at its core. The abstract and Section 2.6 describe an LLM judging the semantic relationship between prompt, response and reference, with a Gaussian prior on a continuous latent quality variable. Section 4 instead trains an amortized network that outputs a Beta posterior, using a uniform prior by default, and Algorithm 1 uses closed-form updates or MCMC with no network at all. In the amortized version, the set encoder in Eq. 11 averages the per-item embeddings, so it never sees the number of items N. For binary scores its input is just the accuracy. A model that gets 7 of 10 items right and one that gets 700 of 1000 right therefore receive the same posterior, although the exact posterior is about ten times narrower in the second case. Measuring how much an evaluation set tells us about performance is the point of the paper, so this undermines the main contribution. I also don't see why amortization is needed, since the exact posterior is closed-form for binary scores and a cheap one- or two-dimensional integral for the Beta likelihood.
Questions for authors
A few clarifications could change my assessment. The most important is whether any experiments were run. The abstract and the contribution list describe results, but the paper contains none, so if results exist, please report the models, benchmarks and numbers. I would also like to know which of the three formulations is the actual contribution: the Gaussian latent-quality model with an LLM judge in the abstract and Section 2.6, the amortized Beta posterior of Section 4, or the conjugate and MCMC procedure of Algorithm 1. How does the encoder in Eq. 11 take the number of items N into account? As written it averages over items, and I do not see how the posterior can then narrow as N grows. Finally, the comparisons in Sections 4.9 and 4.10 treat the models' posteriors as independent. How do they handle the standard case where all models are scored on the same items?