acceptodds
Under review as a conference paper at ICLR 2027

When Agreement Labels Are Necessary for Worst-Group Model Selection

Abstract

Labels on points where all candidate classifiers agree cancel from average-risk comparisons, yet can determine the minimizer of worst-group error. We characterize when this agreement information is necessary for selecting among fixed binary classifiers. On a finite audit pool, a closed-form certificate gives the exact largest regret over a product of label-count intervals and supports adaptive querying and stopping through finite-population confidence sequences. Once all disagreement labels are known, an criterion determines whether a candidate meets the regret tolerance under every completion of the agreement labels. An indistinguishability bound establishes when no uniformly reliable agreement-free selector exists. Matching oracle bounds show that a second group can eliminate the quadratic label savings of small disagreement mass. CAWS uses the certificate with disagreement-first sampling and agreement fallback. Fixed synthetic, digit, Adult, and CelebA pools illustrate both sides of the criterion: four of ten CelebA banks exhibit an agreement obstruction. Matched-path ablations isolate the reduction in certification cost obtained by preserving shared losses.

Primary area:
learning theory
Keywords:
model selection, worst-group risk, active learning, confidence sequences, sequential certification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qEz6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The authors study an "active model selection" problem in which the goal is to pick from a set of candidate models a single model optimizing some objective, by selectively querying the labels of a limited number of datapoints. The authors focus on the objective of worst-group loss.

The paper's main observation is that under a standard objective such as minimizing overall loss, a key part of an optimal querying strategy is that it is unnecessary to query the labels of points where all candidate models agree, as this will not change their relative ranking. In contrast, this is no longer true if the objective is worst-group risk, as the performance on these points may not uniformly affect the worst-group loss metric. Building on this observation, the authors seek to understand how many queries to these "agreement points" are actually needed to select a model that minimizes worst-group loss.

They provide a characterization of a "certificate", i.e. a set of queried points that ensure the approximately best model has been identified with high probability. They analyze the degree to which agreement labels are necessary for certification, and propose an algorithm (CAWS) that actively queries points to select a model while trying to minimize the number of agreement points that are queried. They implement their algorithm and measure its performance on a variety of datasets.

Critical strengths
  • I think the paper's motivating observation is an interesting point, e.g., agreement labels become important for alternative objectives such as worst-group loss.
  • The paper provides both theoretical results as well as empirical evaluation of their proposed algorithm.
Critical weaknesses
  • My primary concern is that the paper is not written clearly, and it was extremely hard to follow and parse. There are a number of terms either only introduced in the Appendix, or never introduced at all, and it was quite difficult to understand the central problem setting of the paper. I think all parts of the paper would greatly benefit from a revision for better clarity.
  • From a motivational standpoint, I am not clear on why minimizing the number of queried agreement labels is an important goal, and would have liked more discussion on why this is the objective for the paper. Is there a reason why agreement labels should be more costly to query than disagreement labels? If not, why not try to develop methods that select the best model while minimizing the total number of labels queried?
Questions for authors

See my second weakness above.

Reviewer_L2EUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper characterizes when agreement labels (labels that all comparator models have made the same prediction on) are needed in order to perform worst-group selection. The authors give a closed-form characterization for binary labels. They define a method (CAWS) to perform worst-group selection which samples first from agreement points and then disagreement points if necessary.

Critical strengths

The question being studied--when the labels in the agreement regions are necessary--is a clean and natural one.

Critical weaknesses

It might have been that my expertise is somewhat outside this area, but I found the exposition to be very dense and difficult to parse. I had to read the first sentence of the abstract multiple times in order to understand it, for example. Generally in the age of AI I feel that much attention should be paid to presentation, and I did not feel like the authors did so here.

Questions for authors

None

Reviewer_9dxuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to choose among already trained binary classifiers when the goal is to minimize worst-group error. It shows that labels of examples on which all models agree can affect the optimal choice, even though their losses cancel in pairwise comparisons within each group. For a fixed evaluation pool, the authors derive an exact worst-case regret certificate over a confidence region for the unknown label counts. They also give a necessary and sufficient criterion for whether a candidate remains near-optimal under every completion of the agreement labels, once all disagreement labels are known. The authors also show that needing agreement labels can substantially increase the labeling cost. CAWS queries disagreement examples first and agreement examples if needed. It stops when it can certify that a model’s regret is within the chosen tolerance. Experiments on synthetic data, digits, Adult, and CelebA test when agreement labels are needed and how many labels are required to certify a selection.

Critical strengths
  • I think the paper asks a focused and relevant question and answers it with an adequate set of theoretical results. The theory seems correct and well-scoped, and the experimental methodology is careful.
  • The paper gives a precise criterion for when agreement labels can be skipped. Once the disagreement labels are known, the criterion determines whether a candidate remains within the chosen regret tolerance for every possible assignment of the agreement labels.
  • The theoretical results explain both how to certify a selection and why additional labels may be unavoidable. The regret certificate provides a valid stopping rule, while the lower bounds show that agreement labels can be necessary for any reliable selection procedure.
Critical weaknesses
  • Thin real-world evidence and trivial acquisition. Obstruction is shown only on CelebA, with the protocol missing. When agreement labels are needed, CAWS matches or trails Uniform, so the practical takeaway beyond the diagnostic is unclear.
  • The main real-data evidence is difficult to reproduce. CelebA is the only natural dataset in the experiments where agreement labels are required, but the construction of its candidate banks is not described in enough detail. Missing: exact model configurations, training and audit splits, checkpoint selection rules, and per-bank correctness results.
  • I find the paper to be unnecessarily difficult to read; from the beginning, it does nothing to help the reader understand its arguments. I usually read papers first and then discuss them with AI, but in this case I had to use AI to check whether I understood all the terms well enough. The authors use symbols before introducing them. Even the paper's main idea (which is very simple) is hard to grasp from the abstract and introduction, and Figure 1 is confusing and uses symbols that are introduced later; it's not immediately clear what the colored dots mean.
Questions for authors
  • Please provide more details about CelebA experiment: Can you provide the exact candidate-bank configurations? Were the banks and their inclusion rules fixed beforehand? For each bank, can you report group sizes and disagreement masses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.