- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of evaluation under a hard budget. In particular, the paper considers the scenario that the learner runs a model with seeds on tasks, in total runs, to determine the main performance. However, the learner have only a limited budget of running evaluations, such that they can only run of the total runs. The paper aims at determining the least budget required for an accurate enough estimation, on the assumption that, within the same task, running with different seeds generally yields similar results. The paper shows that, under proposed strategy, the expected width is roughly if at least one run in every task is finished and when omission is allowed. The method is also verified empirically.
Critical strengths
- This paper focuses on repeated evaluation with fixed budget, which might be of broad interest as running such experiment is becoming costly given the larger models
- The paper proposes a repeat evaluation strategy that leverages the structure that the result within the same task tends to agree, therefore largely reduces the samples required.
Critical weaknesses
I found the paper very difficult to follow. Key terminology and notation are frequently introduced before they are defined, and the presentation does not clearly separate the motivating application, the mathematical setup, and the theoretical objectives. My specific concerns are as follows:
- Abstract: The variable is not defined, and the term “pure cohort” appears without explanation. Both are essential to understanding the main results and should be introduced explicitly.
- Introduction, second paragraph: The phrase “clears a qualification threshold” would be clearer as “exceeds a qualification threshold.” The terms “lower endpoint” and “upper endpoint” appear before the authors explain that the evaluator produces a confidence interval for the score, rather than only a point estimate. Similarly, “fixed cohort” is used without a clear definition.
- Table 1: The table relies heavily on technical jargon, making the comparisons difficult to interpret. I would expect such a table to compare assumptions and results explicitly, for example, whether the present work requires stronger, weaker, or incomparable assumptions, and what guarantees it obtains. Instead, the second and third columns provide compressed descriptions of the settings and inferential objects considered in each work. These descriptions are too brief to be self-contained and do not make the substantive differences clear.
- Problem formulation in Section 3: The setup would benefit from a more precise mathematical description of the underlying objects, observation model, and evaluator’s output. In particular:
- Lines 126–128: The term “path” is introduced and described as containing at most responses or episodes, but its mathematical representation is unclear. The authors should specify what constitutes a path, what information each observation reveals, and how the terminal binary label is determined.
- Line 128: “Decisive event” is not explicitly defined.
- Line 136: The meaning of advancing a path “from observed depth to ” should be defined. The associated cost should also be introduced formally, including what one unit measures and how the stated cost follows from the observation process.
- Line 143: appears without an explicit definition of either and .
- Opening of Section 4: The section begins directly with Theorem 1, without first clearly specifying the class of sampling policies and estimators covered by the theorem or explaining why this lower bound is relevant to the paper’s main objective. A short paragraph stating the statistical problem and motivating the result would improve readability.
- In addition, “MSE” should be expanded to “mean squared error” at its first occurrence.
Given the current status, I believe that the manuscripts requires a substantiative revision before get published
Questions for authors
See weakness.
2 comments
I thank the authors for the efforts of fixing the issues and adapting my suggestions. Since the new version incorporates substantial revision, it does take some time for me to re-evaluate the manuscripts. I will provide additional feedback and / or adjust my rating accordingly if appropriate.
Author Response: Definitions and Presentation Revisions
Thank you for the detailed comments. We revised the abstract, introduction, related-work comparison, problem setup, and theory section so that the evaluation setting is defined before the main results.
- t and “pure cohort.” The abstract now defines M tasks, L planned binary outcomes per task, and a maximum cost of K responses or episodes per run. The budget (M+t)K provides t full-run equivalents beyond one run per task. A pure cohort is defined as one in which the planned outcomes agree within each task. The abstract also states that the target is a confidence interval for the mean of all ML prespecified outcomes.
- Threshold, interval endpoints, and fixed cohort. We replaced “clears a qualification threshold” with “exceeds a threshold.” The introduction now uses a concrete example with 880 tasks and five planned runs per task. These 4,400 prespecified task-run pairs form the fixed cohort. With a budget for 1,100 observations, the evaluator estimates the fixed-cohort mean and reports a confidence interval. A lower endpoint above the threshold supports advancing the model, an upper endpoint below it supports holding the model back, and an interval crossing it indicates that more evidence is needed. We also distinguish this fixed-cohort target from generalization to future tasks or runs.
- Related-work comparison. We revised Table 1 and the surrounding discussion to compare the inferential object or guarantee established by each related work with the additional object studied here. The text now distinguishes expected from pathwise hard budgets, population from fixed-cohort inference, terminal-outcome sampling from partially observed paths, and validity guarantees from matching confidence-width rates. Because some works use different estimands or budget guarantees, we state which aspects are comparable rather than imposing a single stronger/weaker ordering.
- Observation model. Section 3 now defines the objects, observations, costs, and output explicitly. The fixed cohort contains M tasks and L planned paths per task, so N=ML. For path r of task i, Tir∈{1,…,K,K+1} is the first decisive-event index, where K+1 means no decisive event, and Yir=1{Tir>K} is the terminal label. The text explains both first-success and first-failure interpretations.
Advancing an unresolved path from depth a to b costs min(Tir−a,b−a) additional units; one unit is one model response or completed agent episode, and observed prefixes are retained. We use D for the data revealed by the policy and R for its internal randomization. The evaluator returns I(D), with coverage
RPr{θC∈I(D)∣C}≥1−αfor every fixed cohort C. 5. Motivation and organization of the theory. Section 4 is now titled “How replication controls uncertainty” and begins with a roadmap. It explains that incomplete paths cannot always replace completed outcomes, one run per task cannot reveal whether unobserved repetitions agree, and replication supplies this missing information. It then compares preserving coverage of every task with omitting some tasks to fund more replication. Before Theorem 1, we explain why late decisive events form the relevant worst case. Before the one-run-per-task theorem, we explain how a point estimate can be exact while its uncertainty remains poorly certified. The task-covering and omission results are similarly introduced through the allocation decisions they answer, and each main theorem is followed by a plain-language interpretation. The relevant estimands, policy classes, and budget guarantees are defined before use. “MSE” is expanded as “mean-squared error (MSE)” at first occurrence. The empirical section is now organized around three questions: whether Joint improves precision at the same budget, why it improves, and where the gains weaken. We thank the reviewer for these suggestions; the revision incorporates the requested definitions and structural clarifications.