- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces certified stopping for active model selection. Given a budget in model selection it is not clear whether enough labels have been collected to identify the winner. Because of that the paper characterized completion certificates, that is, observed labels that give a guarantee of the unique best model for the remaining labels. The paper shows 9.39% and 10.50% label savings over SELECT, and 26.66% and 30.19% over Model Selector.
Critical strengths
- The paper is well motivated, and the use case is real and relevant
- Along empirical results the paper provides theoretical results
Critical weaknesses
-
Missing baseline comparisons: The paper should compare against uniform random sampling and a recent method such as CODA.
-
Unclear scope and evaluation: The paper does not clearly explain which tasks and evaluation metrics the method supports or what assumptions it requires.
Questions for authors
Clarity
What are the limitations in the general setting? Does the method apply only to classification, or does it also support open-ended classification? Are there any assumptions about the metric function used to evaluate the outputs?
What is a panel? The writing does not clearly explain this term.
The paper consistently references figures and tables in the appendix and presents their numbers as if the results were included in the main paper.
Soundness
How does the metric evaluated here compare to the one used in the Model Selector paper (https://arxiv.org/abs/2410.13609)? Is it the same as identification probability? Could you clarify this with an example?
Could you also provide a comparison against a Random baseline (just randomly sampling data points from a uniform distribution)?
Could you provide a comparison with a more recent baseline, such as CODA (https://arxiv.org/pdf/2507.23771v1)?
2 comments
Thank you for the rebuttal and additional experiments. I appreciate the new baseline comparisons and clarifications. However, I remain concerned about the presentation: even after these exchanges, I still find the formulation, and evaluation difficult to follow. While the rebuttal addresses several of my questions, it does not sufficiently resolve my concerns to support acceptance. I therefore maintain my rating.
Thank you for your careful review. We completed the Random and CODA comparisons under the same exact stopping rule: FrontierCut has lower median label counts than uniform Random across all 19 pools and official CODA acquisition on the three prespecified tasks. To interpret this saving, we distinguish certifying the fixed-pool winner from identifying it correctly in repeated trials.
1. Random and CODA under the same guarantee
FrontierCut's median certification cost is lower than Random's on all 19 pools and CODA's on the three specified tasks. Random uniformly samples unqueried disagreement items without replacement, giving it the same free exclusion of items on which all candidates agree. Across 19 pools with eight runs each, FrontierCut saves a median 40.6% of labels, taking the paired median within each pool and then across pools. All 19 pool-level effects favor FrontierCut. This Random baseline is the explicitly named version of the original “quotient uniform” control.
For CODA (Kay et al., 2025), we used the official acquisition implementation with default parameters and retained every original model and item when initializing its consensus prior. Hard predictions were encoded as one-hot vectors, matching the information available in our cached data. CODA's own prefilter selects query candidates, and the common certificate determines stopping and the returned prediction-equivalence group. The two smallest binary pools and smallest multiclass pool were selected before these comparisons. Median label counts are:
As the table shows, FrontierCut reaches the same certificate with 7.1%, 23.1%, and 33.6% fewer labels than CODA acquisition on WNLI, RTE, and Emotion, respectively. These are certification costs on these three tasks. Each has eight seeded runs, although WNLI and RTE each produce only one distinct CODA query path. Thus repeated identical paths are not additional independent evidence.
2. Supported tasks, metrics, and assumptions
The guarantee applies to finite categorical predictions scored by shared, preset nonnegative item weights: Sa=∑iwi1[pia=yi]. Unit weights recover accuracy. Acquiring one gold label evaluates every candidate on that item, and acquired labels are treated as final. The main stopping experiment requires a unique full-pool winner. Item independence and calibrated probabilities are unnecessary because the guarantee holds for every remaining label assignment.
This includes binary and multiclass classification and LLM tasks whose outputs have a fixed categorical interpretation, such as multiple-choice answers. Unrestricted open-ended generation with semantic similarity or a non-additive metric requires a different score model. For multiclass predictions, the gold answer may match neither of two candidates. Its pairwise contribution is then zero, which our certificate handles explicitly.
3. Relationship to MODEL SELECTOR's identification probability
The metrics answer different questions: we measure labels needed for exact certification, whereas MODEL SELECTOR, Section 4.3.2 measures the fraction of trials identifying the true winner at a fixed budget. For example, selecting the true winner in 95 of 100 runs gives 95% identification probability. In a particular run, however, a lead of three correct answers with five unseen disagreements still permits a reversal. A lead of six with five such disagreements certifies the winner against that rival. Our criterion requires this condition against every rival, and we measure how many labels each acquisition rule needs to reach it.
4. Meaning of “panel” and result presentation
A panel is one candidate collection evaluated on the same fixed item pool, with its cached prediction matrix. An organization panel groups models from one organization. Each reported label count therefore refers to one candidate collection reaching the common certificate on its pool.
The explanation above connects the decision target to its exact certificate and then to label savings, stating the comparisons directly. This makes the argument self-contained. Supplementary tables supply further detail.