- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies best of N search where a verifier scores alternatives, and where a selector, who selects based on the verifier's input, cannot access target labels; in particular, the selector is given a budget, and has to select from a uniformly random subset of the alternatives that fits the budget. As such, in this setting, one considers risk curves as a function of the budget, where the risk comes from the verifier mis-ranking the alternatives compared to the unseen target scores. Properly estimating this risk frontier demands care, due to the dependencies inherent in the estimands, and the paper proposes a way to do so in a practical way, as well as proves certain characterizations --- such as expressions for the expected risk in terms of relevant quantities --- and also provides a real-data empirical playground to test the risk estimation approach.
Critical strengths
I do not see any critical strengths of note that should be highlighted here. The paper is written in a way that complicates the task of reading and reviewing it as much as possible (read: slop), while not giving any strong contributions that would significantly enrich the literatures on best of N, or on risk control. In fact, just reading the list of the 4 contributions in the introduction may convince one of both above points. More on these below.
Critical weaknesses
The paper, as it is written, is, to a very substantial degree, AI slop. Reading the manuscript took inordinately long and was incredibly tedious, and consulting the appendices to try to clarify some of the chaotic terminology didn't help matters. This is the paper's biggest weakness (see below), for which reason I believe it's a clear reject at this point.
Secondly, I am not impressed with the theory, or indeed any of the contributions, proposed by the authors. For example, Proposition 2, declared in the Contributions overview as one of the 4 main contributions, is just a simple combinatorial observation; Proposition 1 is similarly unimpressive/trite. The "finite-sample calibration" contribution is completely mundane/rote too --- of course you need it to apply a method such as the proposed one, and of course you can do it. The "operational audits and limits" contribution, meanwhile, is too unfocused and hard to parse to count as good empirical evaluation. Zooming out into methodology, the literature on risk control, and on controlling multiple risks, is already quite rich and this paper is not contributing much of anything useful to it either.
Getting back to the AI slop discussion, already the abstract, and then the intro, show a near-complete disregard for proper academic writing --- or, indeed, for the basic goal of educating the reader about the substance of the paper. The reader is immediately inundated with obscure terminology such as "frozen bank", "matched-margin" "target-worse", "complete score-gap", "PDE trajectory" (just note this one for instance: how is the reader supposed to tell from reading the abstract ---- perhaps PDE means "partial differential equation" or perhaps "prompt driven exploration"?) . Immediately, unhelpful qualifiers such as "without a Gaussian assumption" spring up, as well as vague concepts like "candidate-wise verifier reliability". Then, instantly, the abstract switches to reporting numerical forecast errors and budget allocations (!) in some particular experiments that the paper runs.
Much the same story then continues in the introduction. Here, once again, instead of gently introducing the area of inquiry or at least concretizing the setting a little bit more, more of the vague language is put in that cannot be parsed by a reader who hasn't internalized the rest of the paper already. More vague terms like "candidate-wise crossing probabilities" and "harmful mask" are introduced, along with further useless and unreadable qualifications such as: "Quadratic-Gaussian geometry provides a controlled illustration, not an assumption of the rank functional". This is then followed by a generated Figure 1 that is almost as uninformative as it could get, with vague allusions to "aligned measurements", "upper envelopes", etc., and with graphics that make little sense on their own.
The same disregard for effective and proper writing continues on through the end of the paper. Now here is the surprise: "PRESCALE", which is in the title of the paper and features throughout the exposition, is actually never formally defined! Not as an algorithm environment, not as an enumerated recipe --- nothing at all. In fact, it is not even made clear to the reader what kind of object PRESCALE even is --- a method? an algorithm? a paradigm? something else entirely? I hope it is now clear where my evaluation of this paper comes from.
Questions for authors
The authors should feel free to respond to any of the above, but I do not currently foresee what could change my evaluation to more positive. This manuscript should have been de-slopped before submitting to a top-tier conference, and ICLR's built-in opportunities for the authors to revise their manuscript will not be appropriate here, as virtually the entirety of the paper needs to be revamped.
2 comments
Thank you for your responses to me and the other reviewers, clarifying certain aspects of the paper's setup/contribution, and for the revision. I have carefully read these. Major presentational issues remain (I don't think the revised manuscript would be clear enough if I was reading it for the first time), so further major revisions (that would be too substantial to handle within this ICLR's reviewing process) targeting clarity will be necessary to potentially turn this paper into a competitive submission.
Response to Reviewer boRj
Thank you for reading our paper. We will improve on our presentation, yet we disagree with the dismissive comment describing the work as "AI slop."
AI helped us refine the writing and assisted with methodology, derivations, code and analysis, as disclosed on p. 10. The authors take responsibility for the research decisions, experimental design and execution, and verification of the results.
PRESCALE and the terminology. PRESCALE is an offline risk-auditing procedure. A "frozen bank" is a fixed candidate pool; the "incumbent" is the current answer. The question is how likely further search is to replace that answer with a higher-scoring but worse one. Section 3.1 (p. 3) now gives the procedure explicitly: use scores and offline target labels to count subsets in which a worse candidate wins, under uniform sampling without replacement. Labels are used for auditing, not by the selector.
The contribution. The counting and concentration tools are classical. The contribution is their use in an audit that accounts for competition between better and worse candidates. A wrong answer may outscore the current answer yet lose to another correct answer. Proposition 2 computes the harmful-winner probability exactly over candidate subsets. Proposition 1 shows why individual crossing probabilities cannot determine it. Statistical calibration is a separate step, requiring independent complete scoring observations on the same pool. We have made this distinction explicit in the revised contribution paragraph.
Why PDE appears. PDE means partial differential equation. The known equation provides structural correctness labels independently of the residual-based verifier. This lets us hold individual crossing probabilities fixed, change score dependence, and measure its effect on selection risk. MATH-500 checks the rank calculation using recorded PRM scores and tests transfer to new pools. It does not validate the PRM's latent error geometry. The revised opening and transitions in Sections 4 and 5 explain why these experiments are needed.
PRESCALE calculates harmful replacement risk exactly over candidate subsets and uses independent repeated complete scoring observations for statistical calibration. We have clarified this procedure and why each experiment is needed. The revisions are marked in blue.