- Rating
- 1
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors study a setting where an annotator, given a fixed set of data points (called a “cohort”) with predictions from two different models but missing ground-truth labels, wants to determine which model performs better efficiently. To do so, they can acquire ground-truth labels sequentially. Their goal is using the models’ predictions and the labels acquired so far to decide which point to label next and when to stop. The main result says that an adversary can force any deterministic adaptive policy to inspect every decision-relevant example.
Critical strengths
- The problem motivation is strong and it uses a useful distinction between adaptive stopping and adaptive reordering. It could be a good contribution given a cleaner exposition and evaluation.
Critical weaknesses
-
The paper is almost impossible to parse as an self-contained manuscript. It contains a lot of jargon even in the Introduction and Problem Set Up section where one expects to be exposed to the technical problem in a clean and coherent order. That makes it very hard to understand the main question it answers and being able to evaluate it for its technical contributions.
Examples: The very first sentence of the introduction: A candidate model is ready to replace the deployed control.
What is this sentence serving? What is a control?
The very first sentence of Problem Setup and Identification is Consider a fixed control and a fixed candidate evaluated on n cases from a fixed cohort. What is the control ? Is it a function? What is the domain and what is the output? -
The theoretical novelty is unclear. The paper should distinguish which results require genuinely new technical arguments from those obtained by combining or extending established results, and explain why these extensions constitute a substantive contribution.
-
In the TREC-COVID analysis, prior-guided adaptive selection improves expected labeling cost over the optimal fixed order under the same prior, but the improvement is minor. This limits the empirical evidence for a substantial practical benefit from adaptive reordering
Questions for authors
- The main theoretical comparison concerns worst-case labeling cost. However since this problem is of practical interest, the implications need clarification: how should a practitioner use this result to decide which examples to label under a limited budget?
- The Cartesian-support assumption does not itself exclude statistical correlations between labels; (it excludes hard cross-example constraints that make some combinations of individually feasible outcomes jointly impossible), can you explain how predictive correlations could inform label selection, even when the feasible support remains Cartesian?