acceptodds
Under review as a conference paper at ICLR 2027

How Many More Labels Are Needed? Minimax Evaluation under Selective Labels

Abstract

Good observed accuracy need not establish acceptable prediction risk when difficult cases lack labels. How much can an additional audit establish? For a frozen cohort with outcome-dependent missingness, we characterize the exact fixed-query asymptotic minimax power over adaptive audits using the full observed record. A standard residual rule attains the truthful frontier with finite-sample false-alarm control. A second result identifies which risk guarantees preserve information in existing observations: count-tail and cumulative predictable-risk guarantees have the same truthful frontier but different noisy critical experiments. The latter reduce auditing to query-only testing, even with independent heterogeneous errors; the former can retain native information. Above criticality, we derive the exact robust frontier with uncertain conditional label reliability, including its blind region, and invert it to quantify the query cost of label-quality uncertainty. This separates three resources: query count, knowledge of label quality, and cohort size. Under count-tail risk bounds and truthful labels, finite certificates show that at 10% versus 20% risk and 5% false alarms, 57 queries suffice and are necessary for 95% power among 1,000 predictions with 500 pending labels. At 12% risk, even complete labels require at least 2,645 predictions for that power. These are scoped design examples, not a general finite-budget formula or validation of deployment assumptions.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
selective labels, statistical model evaluation, minimax testing, auditing, label noise, finite-population inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oymkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the performance of Machine Learning models when there are missing labels in examples (X,y) that are considered hard or important to the performance of the model. The authors derive two theoretical results, 1) under a fixed covariates set (X_1,X_2,...,X_n) with an outcome depedent missing labels mechanism they derive the exact fixed-query asymptotic minimax power over adaptive queries, 2) they examine which risk measures have robust performance under label missingness.

Critical strengths

The authors study an interesting model from an applied perspective and develop theoretical results to quantify the importance of each input data point to the model's accuracy.

Critical weaknesses

As I wrote before, the paper is really hard to read and not even the model the authors work on is clear to me. I ask the authors to give a clear explanation of what is the model they are working on so as the reviewer can quantify the importance of their results. I strongly recommend the authors to write the main body from scratch.

Questions for authors
  1. At the beginning of section 2) (line 97) "Condition on a frozen predictor" -- what is the inference task, what type of predictor is it?

  2. What is the data generating process, are the covariates high dimensional vector arriving iid?

  3. In line 107 what is the difinition of a "randomized kernel"?

1 comment
AuthorsOpenReview ↗

26 hours remaining until score update is closed

Hello, this is a friendly reminder that we have only 26 hours until the score update is closed. Could you reflect your scores based on the renewed manuscript? Thank you.

Reviewer_CzDrOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I have to admit that I fail to understand the setting and objective of this paper.

Critical strengths

None

Critical weaknesses

I think the presentation is so bad that i couldnt fairly evaluate this work.

Questions for authors

None

Reviewer_cSVoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Summary: This paper studies auditing a cohort n items, when a mechanism hides adversarially b of them. We have a budge k of label we can explore among the b that are hidden. The objective is to hypothesis test H_0 : Y_n = E ( V_n | V_{n-1}, .., V_1} is at a for all n, compared to H_1 independent Bernoulis with p>a. For the technique they analyze the super-martingale difference sequence D_n = V_n - a, to get a total high probability estimate for the total cohort, and to analyze the probability for any masking strategy the test the revealed error plus a residual hypergeometric testing for the b pool when seeing k of its elements. They also show tightness in the asymptotic case when a fixed portion of the label is hidden, against any adaptive policy. The audit also controls false alarms at finite n.

Critical strengths

Strengths: The result is interesting covering all angles securing hypothesis testing against an adversarial mechanism, providing guarantees of giving a sharp characterization of testing power, and handling false alarms.

Critical weaknesses

Weaknesses: The techniques of residual testing and the super martingale analysis are known. I would like a clearer motivation for the null assumption E[V_i | V_1, ..., V_{i-1}] ≤ a for every i. A frozen predictor does not by itself imply this bound: if cases share a cohort-level source of difficulty, observing an error can raise the conditional error probability of later cases above a, even when each case has marginal error probability at most a. The paper should explain when the intended auditing setting supports this conditional guarantee.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.