- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies two separate methods for choosing K elements among n candidates. The first is based on ranking expected true quality conditioned on the observed options; this is perhaps the most commonly-used selection rule in practice. The second is to rank options by their probability of being in the Top-K. The authors study when these two policies give the same selected set. They show that when residuals are exchangeable and centered, the two policies are structurally equivalent. In other cases, they show that a disagreement metric depends on how many candidates are different in the two selected sets, and by how much their scores differ from the threshold probability for selection.
Critical strengths
- Interesting topic
- Well-defined question
The question studied in this work is interesting, from a pure curiosity standpoint.
Critical weaknesses
- Unclear writing
- Motivation and assumptions are not sufficiently justified
- Formal statements are not sufficiently defined or explained to understand the precise nature of the result
it was unclear to me why one would use posterior top-K ranking. Do any of the related works you cite consider this policy? Is there any evidence of posterior top-K ranking being used in practice? What motivated you to select and compare to this policy specifically?
I found the paper very difficult to read, possibly due to AI-assisted writing. (my score is not related to whether the paper was AI-written or not, but the unclear writing made it difficult to understand and evaluate the paper). For example, in the introduction, it is not clear what “expected overlap improvement per selected position” means. Improvement over what, and what are selected positions? How do you measure distance from the membership probability cutoff? These terms are defined later, but the introduction needs to be self-contained. I also did not understand parts of several theorem statements. In Theorem 1, what do you mean by “every nonempty relatively open set has positive probability”, and what is the meaning of matrix P? How are sets defined in your statement about relatively open sets? What do you mean by “common-score tie rule”? In Cor. 1. what is the “singular negative endpoint”, and what do you mean by “strict order holds”? What does “only weak order is guaranteed” mean? What is weak order, and what kind of guarantee are you referring to? In general, the “contrast covariance” and P are not sufficiently explained. What are you modeling there?
In Line 218, I didn’t understand why that expectation holds, could you provide some more explanation? In Theorem 2 it wasn’t clear to me what assumptions are being made on the form of the ’s, and their tail behavior in particular.
In lines 212-213, I believe the notation may be ill-defined, as is used both to mean the average density at t and also the density of the th object at , so it’s not clear which one you mean in Thm 2 (I assume the former). Moreover, Theorem 2 (and Section 4.2 in general) would benefit from more explanation and intuition. What are you trying to argue in this section, how do the results relate to the 2 policies under consideration, etc.
If I understand correctly, the paper makes a (fairly strong) assumption that the true quality is generated as plus a zero-mean noise term. I think this assumption requires a more formal and detailed justification, it is not clear at all to me why this should be the case. The empirical evaluation was very difficult to read and understand, and seems like it may have been written with AI. For instance, it’s not clear what you mean exactly by “structural equivalence”, the descriptions of experiments and results are convoluted, etc.
The empirical results require multiple draws or a generative model for scores to estimate the top-K membership probability. Neither of these are practical assumptions in practice; it’s not clear to me how one would compute the top-K probability in practice. This circles back to my earlier question about why we should be interested in this specific policy.
Does your analysis have any implications for methods that select K/n candidates using given uncertainty estimates, such as (Heyard et al, “Rethinking the funding line…” 2022), (Goldberg et al, “MERIT… ” 2025)?
Questions for authors
Why did you choose to study posterior top-K selection?
Thanks for clarifying how the analysis differs from previous work. I'm more persuaded on novelty, and also I found the new analysis of papers near acceptance cutoff helpful. I'll discuss the significance further with the other reviewers before settling on a rating.