- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The authors study an "active model selection" problem in which the goal is to pick from a set of candidate models a single model optimizing some objective, by selectively querying the labels of a limited number of datapoints. The authors focus on the objective of worst-group loss.
The paper's main observation is that under a standard objective such as minimizing overall loss, a key part of an optimal querying strategy is that it is unnecessary to query the labels of points where all candidate models agree, as this will not change their relative ranking. In contrast, this is no longer true if the objective is worst-group risk, as the performance on these points may not uniformly affect the worst-group loss metric. Building on this observation, the authors seek to understand how many queries to these "agreement points" are actually needed to select a model that minimizes worst-group loss.
They provide a characterization of a "certificate", i.e. a set of queried points that ensure the approximately best model has been identified with high probability. They analyze the degree to which agreement labels are necessary for certification, and propose an algorithm (CAWS) that actively queries points to select a model while trying to minimize the number of agreement points that are queried. They implement their algorithm and measure its performance on a variety of datasets.
Critical strengths
- I think the paper's motivating observation is an interesting point, e.g., agreement labels become important for alternative objectives such as worst-group loss.
- The paper provides both theoretical results as well as empirical evaluation of their proposed algorithm.
Critical weaknesses
- My primary concern is that the paper is not written clearly, and it was extremely hard to follow and parse. There are a number of terms either only introduced in the Appendix, or never introduced at all, and it was quite difficult to understand the central problem setting of the paper. I think all parts of the paper would greatly benefit from a revision for better clarity.
- From a motivational standpoint, I am not clear on why minimizing the number of queried agreement labels is an important goal, and would have liked more discussion on why this is the objective for the paper. Is there a reason why agreement labels should be more costly to query than disagreement labels? If not, why not try to develop methods that select the best model while minimizing the total number of labels queried?
Questions for authors
See my second weakness above.