acceptodds
Under review as a conference paper at ICLR 2027

Posterior Top-K Selection under Noisy Reviews: Equivalence and Gain Relative to Posterior Mean

Abstract

Selection problems, including conference acceptance, grant review, and hiring, often require choosing exactly candidates from noisy and heterogeneous evaluations. Ranking aggregate scores does not directly answer the decision-relevant question: how likely is each candidate to belong to the latent top ? We formalize this target through the posterior membership probability . Selecting the largest is Bayes-optimal for expected overlap with the latent top- set. Taking this established decision rule as a starting point, we study when it differs from posterior-mean ranking, why it differs, and how much the resulting roster change matters. We show that exchangeable centered residuals yield a structural equivalence regime. For independent posteriors with continuous densities, we derive a boundary approximation under local regularity; its Gaussian case predicts uncertainty-driven reordering. For arbitrary joint posteriors, we further show that the consequence of disagreement depends jointly on how many candidates change and how far those candidates lie from the membership-probability boundary, and establish when this characterization stabilizes in growing populations. Simulations illustrate these theoretical predictions. Our analysis of ICLR 2026 reviews finds identical rosters at the realized cohort capacity, consistent with nearly homogeneous posterior uncertainty across papers. The Jester results show occasional roster differences, but these yield only small estimated gains under the fitted posterior, while evaluation against an independent high-information reference does not establish a clear advantage. Overall, roster changes alone do not determine the value of posterior Top- relative to posterior-mean ranking.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian decision theory, top-K selection, uncertainty quantification, posterior membership probabilities

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5TbeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies two separate methods for choosing K elements among n candidates. The first is based on ranking expected true quality conditioned on the observed options; this is perhaps the most commonly-used selection rule in practice. The second is to rank options by their probability of being in the Top-K. The authors study when these two policies give the same selected set. They show that when residuals are exchangeable and centered, the two policies are structurally equivalent. In other cases, they show that a disagreement metric depends on how many candidates are different in the two selected sets, and by how much their scores differ from the threshold probability for selection.

Critical strengths
  • Interesting topic
  • Well-defined question

The question studied in this work is interesting, from a pure curiosity standpoint.

Critical weaknesses
  • Unclear writing
  • Motivation and assumptions are not sufficiently justified
  • Formal statements are not sufficiently defined or explained to understand the precise nature of the result

it was unclear to me why one would use posterior top-K ranking. Do any of the related works you cite consider this policy? Is there any evidence of posterior top-K ranking being used in practice? What motivated you to select and compare to this policy specifically?

I found the paper very difficult to read, possibly due to AI-assisted writing. (my score is not related to whether the paper was AI-written or not, but the unclear writing made it difficult to understand and evaluate the paper). For example, in the introduction, it is not clear what “expected overlap improvement per selected position” means. Improvement over what, and what are selected positions? How do you measure distance from the membership probability cutoff? These terms are defined later, but the introduction needs to be self-contained. I also did not understand parts of several theorem statements. In Theorem 1, what do you mean by “every nonempty relatively open set has positive probability”, and what is the meaning of matrix P? How are sets defined in your statement about relatively open sets? What do you mean by “common-score tie rule”? In Cor. 1. what is the “singular negative endpoint”, and what do you mean by “strict order holds”? What does “only weak order is guaranteed” mean? What is weak order, and what kind of guarantee are you referring to? In general, the “contrast covariance” and P are not sufficiently explained. What are you modeling there?

In Line 218, I didn’t understand why that expectation holds, could you provide some more explanation? In Theorem 2 it wasn’t clear to me what assumptions are being made on the form of the fnf_n’s, and their tail behavior in particular.

In lines 212-213, I believe the notation may be ill-defined, as fn(t)f_n(t) is used both to mean the average density at t and also the density of the nnth object at tt, so it’s not clear which one you mean in Thm 2 (I assume the former). Moreover, Theorem 2 (and Section 4.2 in general) would benefit from more explanation and intuition. What are you trying to argue in this section, how do the results relate to the 2 policies under consideration, etc.

If I understand correctly, the paper makes a (fairly strong) assumption that the true quality qiq_i is generated as mim_i plus a zero-mean noise term. I think this assumption requires a more formal and detailed justification, it is not clear at all to me why this should be the case. The empirical evaluation was very difficult to read and understand, and seems like it may have been written with AI. For instance, it’s not clear what you mean exactly by “structural equivalence”, the descriptions of experiments and results are convoluted, etc.

The empirical results require multiple draws or a generative model for scores to estimate the top-K membership probability. Neither of these are practical assumptions in practice; it’s not clear to me how one would compute the top-K probability in practice. This circles back to my earlier question about why we should be interested in this specific policy.

Does your analysis have any implications for methods that select K/n candidates using given uncertainty estimates, such as (Heyard et al, “Rethinking the funding line…” 2022), (Goldberg et al, “MERIT… ” 2025)?

Questions for authors

Why did you choose to study posterior top-K selection?

Reviewer_LvGDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the problem of selecting top-k candidates from noisy and heterogeneous evaluations, comparing the optimal posterior top-k and the posterior mean ranking selection strategies. The paper presents theoretical analysis on when the two strategies are equivalent, when they differ, and how much the difference affect the relevant selection metric. The framework is evaluated on synthetic and real-world datasets, showing some degree of agreement between theory and empirical results.

Critical strengths
  • The paper provides thorough theoretical analysis, along with its assumption, on when posterior top-k selection strategy is equivalent to posterior mean, when they differ, and how big the difference is. This could lead to a better design of top-k selection algorithm.
Critical weaknesses
  1. The motivation and actionable insights that can be drawn from this paper are unclear.

The paper studies the difference between posterior top-k vs. mean selection strategies, and argues that (Line 484) disagreement between the two depends on the disagreement mass and depth, which roughly translates to the difference in the predicted top-k members and posterior distribution of their latent scores. While it is a sound argument backed with theoretical analysis, it is unclear what can the readers do with this information, i.e., why should the readers care about the difference between the two? Moreover, posterior top-k is by construct the maximizer of the assumed-to-be-relevant metric, R_D (Line 161-164). Then why shouldn't we just always choose posterior top-k over posterior mean selection strategy?

A discussion about the impact of this comparative study can better position the paper in the field. For example, one could argue that modeling top-k membership probability is more challenging than predicting a scalar score, such that in specific cases, choosing posterior mean strategy can result in better downstream performance.

  1. The presentation of the paper can be significantly improved.

First, the majority of the experimental setups are delegated to the Appendix, making it hard for the readers to make sense what exactly is being measured or tested in the main text. For example, the simulated experimental results in Section 5 opens with

(L308-310) ... Posterior means are available directly from the simulated models, while posterior top-k membership probabilities are approximated from posterior draws. Reported finite-sample overlap differences use full-draw plug-in membership estimates; a separate split-draw calculation provides an additional numerical check.

There is no explanation about what is being simulated, what are posterior draws, what are the difference between full-draw vs. split draw, and so on.

Second, the texts in Section 5 mostly just restate the results shown in the figures and tables with little discussion. For example,

(L319) Across 18 independent Gaussian arrays, the maximum error... ranges from ... to ... (figure 2a). The approximate roster A_v matches Â_p in ... arrays, whereas A_mean differs in ..., with at most ... replacements.

What conclusion can the readers draw from these numbers? Is a maximum error of 1.7 x 10^-2 good or bad? Why the maximum error in Figure 2a falls by 2 orders of magnitude when n grows from 64 --> 1024? Could this be related to Theorem 3?

Third, there are many instances of new terminology being used without sufficient definition or context, telegraphic sentences, and unnecessarily convoluted sentences. This makes some parts of the papers very difficult to parse without spending significant efforts. For example:

(L077) For any joint posterior, our finite-sample mass-depth identity separates the number of affected candidates from the decision-relevant size of the changes.

(L094) In real data, ICLR illustrates near-equivalence under weak fitted uncertainty heterogeneity, while Jester shows small model-conditional gains that are not resolved under independent-reference evaluation.

(L177) The two ranking rules use the same score-tie convention; when latent qualities have atoms, iid random priorities are used to define T_k(q) without breaking permutation symmetry.

(L215) Continuity ensures exsistence; the local positive-density condition below and monotonicity ensure uniqueness.

  1. As far as I understand, (L1267-1271) imply that the paper-level posterior uncertainty s_i used in Section 6 ICLR experiment depends only on the number of reviews r_i, since v_q and v_e are held fixed. This means that a paper that is rated [10,10,0,0] has the same uncertainty as a paper that is rated [8,8,8,8]. Such modelling choice seems unconventional, and raises a question on the validity of Section 6.1 results that depends on s_i being somewhat uniform across papers.
Questions for authors

Based on its current form, the paper needs a major revision in terms of presentation before I can re-evaluate my assessment.

Reviewer_ruFpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors considers two ways to select K candidates from noisy evaluations. The first ranks posterior mean quality, and the second ranks posterior probability of belonging to the true top-K. It finds when the two rules agree, derives a common-threshold approximation for membership probability, and then gives expected advantage overlap based on how many candidates change and how different their membership probabilities. They show this on a couple distributions. They showed the differences yield small model-conditional gains and no clear advantage against independent reference ratings.

Critical strengths

I think the theory and empirical results are solid. I don't see any issues with the proof. The experiments are well constructed. I also appreciate the topic: I also think the equivalence result is nice (theorem 1). I think the decomposition work is really nice and it's nice to explain how the changing selections has little effect when the exchanged candidates have similar membership distributions.

Critical weaknesses

My main issue is the novelty and impact. There are existing overlapping results in previous works and the impact is not so clear to me. For example, https://pmc.ncbi.nlm.nih.gov/articles/PMC2664159 This paper gives very similar results. Another issue is that the practical results are not very strong. The jester largest estimated conditional improvement is only 0.173 per selected position and the significance is generally not shown, though this is noted. The authors explicitly write "Under the independent reference, all eight paired-panel intervals include zero." Also, the theoretical approximation bounds are usually too loose to be useful. For example, the uniform probability-error bound reaches the trivial cap in 2/3 of the simulated settings. Hence, the theoretical guarantees also don't seem to have much practical relevance.

Questions for authors

Can you explain how this work differ or extend from the paper I linked?

1 comment

Thanks for clarifying how the analysis differs from previous work. I'm more persuaded on novelty, and also I found the new analysis of papers near acceptance cutoff helpful. I'll discuss the significance further with the other reviewers before settling on a rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.