- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the winner's curse in best-of-N prompt selection, where reporting the best candidate's dev score overstates its true accuracy. The paper measures how this bias depends on the number of candidates and the size of the dev set, and argues that correlated candidate pools behave like only a few independent candidates. It proposes a correction based on the effective number of candidates. It also offers guidance on splitting a fixed evaluation budget between more candidates and larger dev sets.
Critical strengths
It is interesting to transfer the idea of the winner's curse to prompt selection domain.
Critical weaknesses
-
Limited novelty and contribution. Optimistic bias from selecting a winner on its validation data is well-known. The setting is also narrow and out-of-date: single-token classification with two ≤3B models released in 2024. The contribution offers little new knowledge for current practice.
-
A central claim contradicts the released results. The abstract and conclusion repeatedly claim that two estimators agree on approximately 1.5–3.5 effective candidates. But the paper reports no numerical results for the moment-matching estimator. In the linked repository’s tables/table4_neff.md, its exact estimates at m=100 are 6.3–15.6, which is approximately four to five times the spectral estimates.
Questions for authors
-
Why did you study two small models released in 2024 and only single-token classification tasks? Do your main conclusions still hold on larger, more recent models and on more complex tasks, such as generation, reasoning or even agentic tasks?
-
The abstract and conclusion say both estimators give ~1.5–3.5 effective candidates, but your repository's table4_neff.md reports 6.3–15.6 for the moment-matching one. Could you explain this mismatch?