acceptodds
Under review as a conference paper at ICLR 2027

Not All Mislabels Are Equal: Efficient Task-Aware Auditing with Confidence Sequences

Abstract

We study efficient, adaptive auditing of categorical labels in a fixed dataset, motivated by biodiversity-credit programs in which expert verification of species records is costly and downstream scores (e.g., payment) computed from corrected labels are more sensitive to errors on certain labels than on others. Our framework couples anytime-valid inference with a task-aware adaptive auditing policy, representing label corrections through a confusion matrix and prioritizing which data points to inspect next based on the uncertainty within their class labels and the corresponding influence of these labels on the downstream score function. For linear scores such as the mislabel rate, our priority-aware approach certifies the overall label error rate to +/- 2 percentage points with 24% and 11% fewer audits compared to random sampling on biodiversity data from Kenya and a widely-used CIFAR-10N benchmark, respectively. For nonlinear scores such as the Hill–Shannon diversity, we exploit a linear-envelope representation that reduces inferences to a family of linear-score audits; priority-aware sampling again reaches the target precision with 36% fewer audits. When the cost of distribution-free guarantees becomes prohibitive, we develop a model-based Bayesian alternative that trades assumption-free validity for considerably fewer audits. Together, these results show task-aware auditing can significantly reduce the number of expert audits needed while providing finite-sample, anytime-valid guarantees.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
confidence sequences, label verification, confusion matrix, decision-aware inference, data-aware auditing, biodiversity credits

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_QSuAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the setting where the “ground truth” data (for a finite dataset) used for downstream inference are noisy, and need to be reviewed with limited expert effort. The proposed solution uses a confusion matrix to represent label corrections and prioritizes verification effort on data that most narrow confidence intervals for the downstream task. The authors demonstrate the effects of this approach on using wildlife data for biodiversity credit decision-making.

Critical strengths
  • The problem setting of active review is well motivated, and the proposed method is evaluated on a real-world application where it is clearly relevant.
  • The authors provide strong intuition for assumption-free anytime-valid audits.
  • The paper ends with a discussion of limitations and opportunities for future work.
Critical weaknesses
  • Several citations are missing from the Related Works. Consider adding citations for post-prediction inference [1], and prediction-powered inference (PPI) [2].
  • While I am convinced that this problem setting is important, this argument/perspective is not necessarily novel in itself. This paper [3] is also a relevant example of a paper that does active review of noisy ML outputs for a specific downstream inference target. The language should reflect the fact that other workshave considered a similar setting.
  • The overall framework echoes this method for active model selection [4]. Consider adding a citation to the Related Works, and discussing how they are complimentary.
  • Include standard, non-random-based baselines (a downstream entropy-based sampling, for example).
  • There are several key missing details about the methodology. The datasets are not cited and the downstream goals optimized for are not sufficiently described in the main text.
  • Report results over seeds in the tables for the means and coverages.
  • The discussion of results is dense with numerical results, making it hard to parse the high-level takeaways. The results are not sufficiently discussed in the main text (i.e. the confusion matrix in Figure 2 is not directly mentioned).

[1] Wang, S., McCormick, T. H., & Leek, J. T. (2020). Post-prediction inference. BioRxiv, 2020-01. [2] Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., & Zrnic, T. (2023). Prediction-powered inference. Science, 382(6671), 669-674. [3] Haucke, T., Harrell, L., Kay, J., Clapp, M., & Beery, S. (2026). Targeted Review for AI-Assisted Biodiversity Surveys: Active Continuous-Score Occupancy Modeling. arXiv preprint arXiv:2609.25657. [4] Kay, J., Van Horn, G., Maji, S., Sheldon, D., & Beery, S. (2025, October). Consensus-driven active model selection. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 4594-4604). IEEE.

Questions for authors
  • What differentiates this method from previous active review literature?
  • What determines when an expert should stop reviewing in practice? Is the only stopping condition a constraint on the confidence or relative accuracy? What determines the stopping point for the figures and the tables?
4 comments

Thank you for your thoughtful review. We appreciate your recognition of the problem’s practical relevance, the intuition behind our assumption-free anytime-valid audits, and the discussion of limitations and future directions.

W1–W3 and Q1) Related work and contribution

Several citations are missing from the Related Works.

The overall framework echoes this method for active model selection [4].

What differentiates this method from previous active review literature?

Thank you for the related work suggestions. We revised Section 7 to cite and discuss Wang et al. (2020), Angelopoulos et al. (2023), Haucke et al. (2026), and Kay et al. (2025), acknowledging prior work on both prediction-assisted inference and active review. Our framework additionally certifies the resulting quantity with an anytime-valid uncertainty guarantee. Our contribution is to combine task-aware adaptive review with finite-sample, distribution-free confidence sequences that remain valid under both adaptive sampling and adaptive stopping, including for nonlinear downstream quantities. For a given score, our PAS uses Bayesian predictions of label corrections to estimate how much each record’s contribution varies within its stratum. These estimates guide sampling toward the variance-minimizing allocation characterized in Proposition 3. Shekhar et al. (2023) provides a direct comparison for linear scores because their method supports anytime-valid inference under weighted sampling without replacement. For comparative results against a collection of new baselines, see W4 below.

W4) Non-random baselines

Include standard, non-random-based baselines (a downstream entropy-based sampling, for example).

Thank you for the suggestion. For mislabel rate, we evaluated Prop-MS and Prop-M + CV from Shekhar et al. (2023) using the authors’ released code. Table 1 reports comparisons across four datasets, including the newly added ReImageNet and CleanCoNLL. PAS required significantly fewer audits than Prop-MS on all four datasets and Prop-M + CV on three of four (Table 1).

We also certify nonlinear scores through a linear-envelope representation of entropy and a linear-ratio representation of F1. We added PPR with projection, extending the prior–posterior-ratio method of Waudby-Smith and Ramdas (2020), and adapted the method from Manole and Ramdas (2023). PAS reduces the number of audits required to certify the nonlinear scores substantially when compared against these baselines (Table 8).

To test whether audit selection improves when it accounts for how each possible label correction affects the score, we compared PAS with an adaptive policy based on label uncertainty alone. Across the five nonlinear scores considered, PAS reduced the median audit count by 6%–38% relative to this baseline (Table 8).

W5) Dataset and downstream-task descriptions

The datasets are not cited and the downstream goals optimized for are not sufficiently described in the main text.

Section 4.1 now cites CIFAR-10N, ReImageNet, and CleanCoNLL; Appendix D.1 describes the Kenyan dataset and its source. We also release the Kenya confusion matrix with species names. In our linear-score experiments, the task is to certify mislabel rate to within two percentage points at a nominal 95.5% confidence level using as few audits as possible (Section 4.1). Section 5.1 now describes each nonlinear score and its precision target.

W6) Variation across audit paths and coverage

Report results over seeds in the tables for the means and coverages.

Thank you for the suggestion. All tables now report the median audit count and the 25th and 75th percentiles across 100 audit paths. Figure 10 shows how often the intervals covered the true Kenyan mislabel rate and diversity score through the full audit path while varying initial predictions. Table 9 reports whole-path coverage for all five nonlinear tasks. For the Bayesian procedure, Table 2 reports coverage both at stopping and at every checkpoint through stopping. The supplementary material includes the random seeds used in the experiments.

W7) Clearer results discussion

The discussion of results is dense with numerical results, making it hard to parse the high-level takeaways. The results are not sufficiently discussed in the main text (i.e. the confusion matrix in Figure 2 is not directly mentioned).

We rewrote Sections 4.1 and 5.1 to emphasize performance trends across baselines and direct readers to the appendix for detailed comparisons. Figure 2’s confusion matrix displays submitted-to-verified label patterns in the Kenyan input dataset, from which all methods sample. We clarified this role in the text (Section 2, Section 4.1).

Q2) Practical stopping and reported audit counts

What determines when an expert should stop reviewing in practice?

The auditor can stop when the interval is sufficiently precise for the decision or when the review budget is exhausted. If the budget runs out first, the current interval and achieved precision are reported without claiming that the target was met. For the figures and tables, we record the first evaluated checkpoint at which the 95.5% interval meets an arbitrary prespecified target: for example, a half-width of two percentage points for mislabel rate, or 5% of the interval midpoint for Hill–Shannon diversity. Domain-experts would select these thresholds in practice.

Thank you again for highlighting the practical relevance of our work and suggesting connections to the active-review literature. Your feedback helped us better present our contribution, strengthen the comparisons, and make the experimental goals and results clearer.

Thank you for your thoughtful and detailed response. I have reviewed the updated submission, and I believe that the edits meaningfully improved it. In particular, the revised Related Works section definitely helps clarify the contribution of this work, and the uncertainty-only ablation and additional datasets strengthen the comparisons made in the paper. It is very clear that task-aware review is more efficient than standard uncertainty-based review. Just one more question: since improvements over uncertainty-driven review is variable, it would be interesting to discuss when using more complex task-aware selection is actually justified. This is more a point of further clarification. I am increasing my score from a 2 (weak reject) to a 3 (weak accept).

Thank you very much for your encouraging follow-up. We’re glad the revisions addressed your concerns.

To answer your question: task-aware selection is most useful when labels with similar uncertainty have different consequences on the downstream score. For example, suppose two camera locations have similarly uncertain predictions: one mainly confuses cattle with sheep while the other confuses cattle with lions. Sheep and cattle are both common in the dataset, but lions are rare, and therefore more influential on the downstream diversity score. Resolving the second location’s labels therefore matters more, so task-aware selection can prioritize it even when uncertainty-only selection treats the locations similarly. This illustrates the differences in score-relevant uncertainty discussed in our limitations (lines 532–534).

We appreciate your careful reassessment and increasing your score from 2 to 3. There may be a delay in OpenReview, but so far we do not see the score update yet.

Reviewer_uRN2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies adaptive auditing of categorical labels in a fixed dataset. The dataset is split into strata based on submitted labels and metadata. For linear scores, the authors propose:

  • unbiased estimator, combining a prediction with an inverse-probability-weighted correction,
  • anytime valid confidence sequences using e-process,
  • priority-aware sampling of strata based on Neyman allocation.

The authors extend their approach to nonlinear scores including Hill-Shannon diversity using linear representation. The authors also propose a Bayesian alternative and evaluate the methods on biodiversity data and CIFAR-10N.

Critical strengths
  1. The paper addresses a practically relevant problem and provides anytime valid guarantees under adaptive sampling.
  2. The construction of confidence sequences for entropy using linear envelopes and convex optimization is technically interesting.
  3. The experiments demonstrate improvement in number of audits compared with random sampling.
Critical weaknesses
  1. Novelty. The linear-score approach combines known techniques and the novelty compared to Shekhar et al (2023) and Xu et al (2024) is not discussed enough
  2. The experimental comparison is limited. Most baselines do not incorporate prediction assisted adaptive inference, and the large improvement over PPR combines several components. Moreover, Table 1 reports only median stopping times without uncertainty estimates.
  3. The construction of confidence sequences for worst-three precision and taxonomic risk is not sufficiently explained. Appendix D.2.2 describes the scores and sampling policy but does not establish the anytime-validity.
Questions for authors
  1. How is the floor constrained optimal allocation in proposition 3 related to the policy in equation 7?
  2. What happens if all records have been audited but the stopping criterion of Algorithm 1 is not satisfied?
  3. Lines 172-177. Both ItI_t and JtJ_t are used for record index.
2 comments

Thank you for your thoughtful review. We appreciate your recognition of the problem’s practical importance, our anytime-valid guarantees under adaptive sampling, and the confidence-sequence construction for entropy.

W1) Distinction from related work

The linear-score approach combines known techniques and the novelty compared to Shekhar et al (2023) and Xu et al (2024) is not discussed enough

Xu et al. (2024) selectively request labels to control predictive risk on future examples. We instead verify labels in an existing dataset until a downstream score is certified to the desired precision. Their guarantees concern future draws and do not directly provide anytime-valid inference for fixed finite-population scores under adaptive sampling without replacement.

Shekhar et al. (2023) provide a direct comparison for linear scores because their method supports anytime-valid inference under weighted sampling without replacement. For a given score, PAS uses Bayesian predictions of label corrections to estimate how much each record’s contribution varies within its stratum. These estimates guide sampling toward the variance-minimizing allocation characterized in Proposition 3. We clarified these distinctions in the introduction and Section 7. Using the authors’ released code, PAS required significantly fewer audits than Prop-MS on all four datasets and Prop-M + CV on three of four (Table 1).

We also certify nonlinear scores through a linear-envelope representation of entropy and a linear-ratio representation of F1. We added PPR with projection, extending the prior–posterior-ratio method of Waudby-Smith and Ramdas (2020), and adapted the method from Manole and Ramdas (2023). PAS reduces the number of audits required to certify the nonlinear scores substantially when compared against these baselines (Table 8).

W2) Empirical comparisons and variability

Most baselines do not incorporate prediction assisted adaptive inference, and the large improvement over PPR combines several components.

Table 1 reports only median stopping times without uncertainty estimates.

Thank you for the suggestion. We expanded evaluation to four datasets by adding ReImageNet and CleanCoNLL, alongside the prediction-assisted adaptive baselines described above. We also added three sensitivity studies (Appendix D.8): sweeps over the exploration weight, initial predictions for both mislabel rate and diversity, and predictor misspecification through underconfidence, overconfidence, constant uncertainty, and reversed uncertainty rankings.

The principal experiments now use 100 paths. Tables 1 and 8 report medians and 25th–75th percentiles across paths.

W3) Confidence sequences for worst-three scores

The construction of confidence sequences for worst-three precision and taxonomic risk is not sufficiently explained.

We added the confidence-sequence construction and its anytime-validity argument for both scores in Appendix D.2, including how the reported intervals are obtained.

Q1) Sampling floor and optimal allocation

How is the floor constrained optimal allocation in proposition 3 related to the policy in equation 7?

Under exact predictions, Proposition 3 characterizes the variance-minimizing allocation subject to sampling floors. Equation (7) is a practical approximation: with probability ϵ\epsilon, it samples uniformly from the remaining records. Otherwise, it selects a stratum in proportion to its remaining size times the predicted standard deviation of its score contributions. Each stratum therefore receives at least ϵ\epsilon times its share of remaining records. This mixture preserves exploration and uses estimated variability to guide allocation.

Q2) Exhausting the audit population

What happens if all records have been audited but the stopping criterion of Algorithm 1 is not satisfied?

The audit stops when the precision target is reached or all NN labels have been verified. In the latter case, the downstream score is known exactly and we report the true score.

Q3) Record-index notation

Lines 172-177. Both ItI_t and JtJ_t are used for record index.

Thank you for pointing this out. We now use JtJ_t consistently for the sampled record index, reserving ItI_t for the confidence interval.

Thank you again for your careful review and constructive suggestions. We believe your feedback has helped us clarify the methodology, strengthen the baseline comparisons, and broaden the empirical evaluation.

Thank you for your response and clarifications. I will keep my score.

Reviewer_txhpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies efficient adaptive auditing of categorical labels in a fixed dataset. The dataset already has labels but some of them may be wrong. Expert verification of each record is costly, and the downstream scores are far more sensitive to errors on certain labels than on others. The paper proposes a task-aware auditing framework with three components. (1) A confidence sequence built on inverse-probability-weighted corrected observations and betting-style e-processes. (2) A priority-aware sampling strategy (PAS). (3) For nonlinear scores such as the Hill-Shannon entropy, a linear-envelope (tangent plane) representation that turns the lower-bound certification into a convex optimization. A hierarchical Dirichlet-multinomial Bayesian alternative is also provided in exchange for fewer audits. Experiments are conducted on Kenyan biodiversity data and CIFAR-10N. Certifying the mislabel rate to plus or minus 2 percentage points saves 24% and 11% of audits respectively, and the P2 diversity target saves 36%. The Bayesian version needs 248 audits on average but its coverage of 93% falls below the nominal 95.5%. From the perspective of the research problem, I can roughly understand its research value. However, the structure and the argumentation logic of this paper are messy, which makes it hard for me to confirm the methodological contribution and novelty (see Weakness 1). The experiments are also incomplete (see Weakness 2) and not very convincing, which limits me from giving a better score. This paper still has much room for improvement before it becomes a good piece of work. Unless it goes through a thorough restructuring and revision, I am not inclined to raise my score. I hope the authors will pay attention to clarity and logic in argumentation, as well as the completeness of the experiments.

Critical strengths

S1. The problem setting is valuable. The paper formalizes label auditing as finite-population inference and proves that the predictability of the sampling policy does not affect the validity of the confidence sequence (Theorem 2), so that efficiency optimization (Section 4) and statistical guarantees (Section 3) can be designed independently. This decoupling allows the Bayesian version (Section 6) to reuse the same sampling policy. S2. The synthetic mechanism study provides attribution evidence. By constructing a controlled population with score-sensitive rows and score-invariant rows, the paper verifies that the gain of PAS grows with the proportion of irrelevant rows according to 1 - sqrt(k/R), which also confirms the applicability boundary of the method. This supports the causal explanation of the method better than a pure dataset comparison.

Critical weaknesses

W1. The paper is not systematic, and the presentation weakens the completeness and readability of its contributions. Four points are listed below.

  1. Motivation, statistical tools, and efficiency claims, three different kinds of information, appear alternately in the first three paragraphs. The core questions are hidden in the middle of the statistical survey. Although the first paragraph raises two questions, I cannot map them well onto the text of the argument. The problem definition feels like giving the shape of the answer first and the question itself afterwards. This reads as a premature definition.
  2. Figure 1 contains three subfigures and a large amount of text annotation. The text density is high, the font size is small, and the pipeline has to be assembled by readers themselves. In my view, it fails to serve as an overview and does not let me directly understand the design of the method.
  3. The methods and the experiments are typeset alternately, which cuts a unified framework into three half-finished pieces each followed by its own experiments. The paper does not present the proposed components as an integrated framework. The PAS configurations of the three experimental groups differ from each other, and the scattered typesetting makes this inconsistency hard to notice.
  4. Among the three contributions listed in the introduction, the "categorical correction structure" is not substantively used at the theorem level. The theory only requires the contribution matrix D in [0,1] and predictability, while the categorical structure only appears in the predictor fitting and the variance heuristic. W2. The experimental setup is incomplete and insufficient to demonstrate the advantage of the method over existing work. Three points are listed below.
  5. The baselines mainly use random sampling, and recent works from 2025 to 2026 are missing. The paper itself describes Shekhar et al. (2023), Zrnic and Candes (2024), Xu et al. (2024), and Li et al. (2025) as close works, yet none of them enters the experimental comparison. All the gains of PAS come from the sampling strategy, and predictable nonuniform sampling is a capability already available in Shekhar et al. (2023). Without this comparison, the increment of the core contribution cannot be substantiated, and it is also unclear whether this problem has been effectively solved in recent years.
  6. The experiments use only two real datasets, one of which is confidential and cannot be reproduced, plus one synthetic study. There is no generalization across multiple datasets and no sweep over key conditions. The characterization of the applicability domain of the method relies on a single synthetic experiment.
  7. There is no parameter sensitivity analysis and no ablation study, so the core claim that the task-aware objective is necessary lacks direct evidence. It is worth noting that the authors have already evaluated six candidate configurations on CIFAR, so the sensitivity data most likely exist but are not reported.
Questions for authors
  1. Please clarify what the research motivation and the research goal of this paper actually are, or explain the necessity of the current way the introduction is written.
  2. Please point out the exact place where the "categorical correction structure" is used in Lemma 1 or Theorem 2. If the theory only relies on D in [0,1] and predictability, what is the theoretical increment of this paper over Shekhar et al. (2023)?
  3. The variance optimality of Proposition 3 holds under the assumption of exact predictions (b = 0). When predictions are biased in practice, is there any theoretical characterization or experiment on the variance advantage of PAS over SRS?
  4. Please add recent works as baselines, implement them, and compare, and also increase the number of datasets. If the latest works cannot be experimented with, or if all other datasets are inapplicable, please explain why this comparison is infeasible.
  5. Please add the necessary basic experiments, such as ablations and hyperparameter sensitivity. The exploration weight epsilon differs across the three experimental groups. Please explain the principle for choosing epsilon.
  6. The main experiments use the median stopping time over 20 paths, while the Bayesian and synthetic experiments use 100 paths each. Please report the dispersion of the median in the main experiments, or explain why the numbers of paths are inconsistent.
3 comments

Thank you for your thoughtful review. We appreciate your recognition of the problem’s importance, the separation of sampling-policy optimization from anytime-valid inference, and the synthetic study isolating why PAS improves efficiency.

W1 and Q1) Presentation, motivation, and goal

The paper is not systematic, and the presentation weakens the completeness and readability of its contributions.

Please also see our general response to all reviewers for an overview of the revisions. We rewrote our introduction and restructured the presentation of central results. Our original motivation is to audit species labels in biodiversity datasets, where expert verification is expensive. Our goal is to develop a general efficient, anytime-valid procedure that adaptively selects which records to verify and allows the audit to stop as soon as the downstream score is certified to the desired precision.

W1 and Q2) Categorical structure and theoretical scope

Please point out the exact place where the "categorical correction structure" is used in Lemma 1 or Theorem 2.

Thank you for this careful observation. For a fixed linear score, Lemma 1 and Theorem 2 require bounded numerical contributions and predictability, not the full categorical correction structure. Retaining categories allows the same predictor to support different scores: correcting species A to B or C counts equally toward mislabel rate but can affect a species-weighted score or diversity score differently. PAS uses predicted category probabilities to estimate this score-relevant variation within each stratum and guide sampling.

W2 and Q4) Baselines, datasets, and reproducibility

The experimental setup is incomplete and insufficient to demonstrate the advantage of the method over existing work.

We added two public datasets ReImageNet and CleanCoNLL, expanding evaluation to four settings. We also release the Kenya confusion matrix with species names.

For mislabel rate, we implemented Prop-MS and Prop-M + CV using Shekhar et al.’s released code. PAS has a lower median audit count than Prop-MS on all four datasets and Prop-M + CV on three. On CleanCoNLL, Prop-M + CV has a lower median but reaches the target on only 75 of 100 paths within a budget of 5,000 audits, compared with all 100 for PAS (Table 1).

For nonlinear scores, we added PPR with projection, extending the prior–posterior-ratio method of Waudby-Smith and Ramdas (2020), and adapted the method from Manole and Ramdas (2023). PAS reduces the number of audits required to certify the nonlinear scores substantially when compared against these baselines (Table 8).

W2 and Q5) Ablations and exploration weight

Please add the necessary basic experiments, such as ablations and hyperparameter sensitivity.

Thank you for the suggestion. We varied the exploration weight across all four datasets (Appendix D.8.1) and tested sensitivity to the initial predictions on Kenya (Appendix D.8.2). PAS was stable across a broad range of initializations: its median audit count varied by less than 5% for both scores. For the mislabel rate, the median ranged from 603 to 631 audits as the initial probability of a correct label varied from 20% to 80%, with higher costs at the 0% and 100% extremes. For Hill–Shannon diversity, the median ranged from 1,550 to 1,625 audits as the initial weight on the uniform label distribution varied from 10% to 100%.

The original submission already compared PAS with SRS using the same predictor and confidence-sequence construction to isolate the effect of priority-aware sampling. To test whether audit selection improves when it accounts for how each possible label correction affects the score, we compared PAS with an adaptive policy based on label uncertainty alone. Across the five nonlinear scores considered, PAS reduced the median audit count by 6%–38% relative to this baseline (Table 8).

Q3) Imperfect predictions

When predictions are biased in practice, is there any theoretical characterization or experiment on the variance advantage of PAS over SRS?

Thank you for the suggestion. We evaluate flat and deliberately wrong initial predictions, as well as systematic predictor misspecification through underconfidence, overconfidence, constant uncertainty, and reversed uncertainty rankings (Appendix D.8). PAS remains efficient under moderate misspecification, while severe overconfidence or incorrect rankings can eliminate its advantage.

Q6) Consistent path counts and dispersion

The main experiments use the median stopping time over 20 paths, while the Bayesian and synthetic experiments use 100 paths each.

Thank you for the suggestion. All experiments now use 100 audit paths. Tables 1 and 8 report median stopping times and 25th–75th percentiles across paths.

Reviewer_txhpOpenReview ↗

No satisfied response received

Thank you for your response and clarifications. But I am not satisfied with your response and I will keep my score.

AuthorsOpenReview ↗

Clarification of Our Revisions and Contributions

Thank you for following up. Our revision seeks to address each point raised in your review (clearer presentation, adding more baselines of existing adaptive anytime-valid methods, two additional public datasets, sensitivity analyses, ablation experiments, and reporting variability in audit counts).

The practical benefit of our method is a substantial reduction in costly expert review. In our Kenyan biodiversity case study, ATLAS-PAS certifies the biodiversity score after auditing approximately 6% of the data, compared with the 50% audited in practice by our partner NGO, reducing expert review by an order of magnitude while guaranteeing anytime validity. For mislabel-rate certification, our method requires only 25% or fewer audits as the evaluated state-of-the-art anytime-valid methods of Shekhar et al. (2023) and Waudby-Smith and Ramdas (2020), at the same confidence level and precision target (Table 1). Across four datasets, our method on average requires over 54% fewer audits than Shekhar et al.’s methods. It also requires the fewest audits among the evaluated methods for all five nonlinear scores. For Hill–Shannon diversity, it reduces median audit counts by more than 83% relative to both PPR with projection, based on Waudby-Smith and Ramdas (2020), and our adaptation of Manole and Ramdas (2023), with substantial reductions in the other nonlinear settings as well (Table 8).

Beyond improving efficiency for linear scores, our work is also the first to develop anytime-valid confidence sequences tailored to nonlinear scores under task-aware sampling, providing new capabilities beyond Shekhar et al. (2023) or existing approaches. We use Bayesian predictions to guide task-aware sampling, and distribution-free confidence sequences preserve validity even when those predictions are inaccurate. We believe this combination of substantial practical audit savings and rigorous guarantees for downstream (linear and nonlinear) quantities provides a valuable contribution to the community.

Reviewer_U4tYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at auditing labels in a fixed dataset when expert checks are expensive and some label errors matter more for the downstream score than others. It samples records adaptively, using known per-stratum selection probabilities with prediction-corrected, inverse-probability-weighted observations, so a betting-style confidence sequence stays anytime-valid (Lemma 1, Theorem 2). For nonlinear scores like Hill-Shannon diversity, it builds upper and lower confidence sequences through linear envelopes (Section 5, Appendix G). On Kenyan biodiversity records and CIFAR-10N, priority-aware sampling (PAS) needs 24% and 11% fewer audits than random sampling (SRS) for the error rate (Table 1), and 36% fewer for diversity. A Bayesian variant needs far fewer audits but under-covers (93% vs nominal 95.5%, Table 2).

Critical strengths
  1. The nonlinear part is the most interesting contribution. The entropy lower bound in Appendix G profiles over all feasible compositions and uses a conservative certificate when the optimizer is inexact, so a bad numerical solve only widens the interval. I went through the main proofs and the validity argument looks right to me.

  2. The PAS vs SRS comparison is clean: same confidence sequence, same target width and level, paired paths (20 per setting), and medians with spread in Figures 2 and 3. The paper is also upfront that the Bayesian version under-covers.

Critical weaknesses
  1. The baselines are all random sampling. Table 1 compares PAS to SRS, Wald, PPR and Serfling, but Section 7 itself names Shekhar et al. (2023) as the closest work and doesn't compare to it. So we know the priority rule beats random sampling inside this pipeline, but not how much it adds over existing adaptive auditing, especially for the linear error-rate case.

  2. I'm not sure how much of the gain comes from priors fitted on the same data. Proposition 3 assumes exact predictions. For CIFAR-10N, the stratum error priors are fit on a "development view" that swaps in random_label2 as the submission (Appendix D.2.1), which seems to use the same 50,000 images and possibly the same clean labels that are later audited. Six policies were also picked on this view. Coverage stays valid either way, but the audit savings could be optimistic. There is also no test with bad or misleading priors.

Questions for authors
  1. For the CIFAR-10N development view, which reference labels were used to fit the error priors, and are those images disjoint from the ones audited in Table 1? (clarification request)

  2. Could you add a run with flat or deliberately wrong priors, and a comparison with an adaptive finite-population method like Shekhar et al. (2023) at the same level and width? (suggestions for future work)

  3. In Section 6, is the Bayesian model a full joint posterior or an empirical-Bayes approximation with an updated pooled prior? This affects how to read the 95.5% credibility claim. (clarification request)

3 comments

Thank you for your thoughtful and constructive review. We appreciate your careful examination of the confidence sequence construction and are encouraged that you found the validity argument sound. We also appreciate your recognition of our thorough comparison between PAS and SRS and our discussion of the Bayesian method’s undercoverage.

W1) Comparisons with related methods

The baselines are all random sampling. Table 1 compares PAS to SRS, Wald, PPR and Serfling, but Section 7 itself names Shekhar et al. (2023) as the closest work and doesn't compare to it.

Thank you for the suggestion. For mislabel-rate certification, we evaluated two variants from Shekhar et al. (2023), Prop-MS and Prop-M + CV, using the authors’ released code (Table 1). We also added two public datasets, ReImageNet and CleanCoNLL. PAS required fewer audits than Prop-MS on all four datasets and fewer than Prop-M + CV on three of four. Table 1 reports medians and interquartile ranges for all four datasets.

For nonlinear scores, we added two baselines. The first, PPR with projection, extends the prior–posterior-ratio method of Waudby-Smith and Ramdas (2020), which we already evaluated for the mislabel rate. The second adapts the method of Manole and Ramdas (2023) to each nonlinear score. Appendix D.5.1 describes both constructions and the required adaptations.

Across all five nonlinear settings, PAS had the lowest median audit count to reach the target precision. Figure 3 shows how interval widths change with audit count, and Table 8 reports the full numerical comparison.

W2 and Q1) Development data and initialization

I'm not sure how much of the gain comes from priors fitted on the same data.

For the CIFAR-10N development view, which reference labels were used to fit the error priors, and are those images disjoint from the ones audited in Table 1?

You are correct: the original development view used random_label2 as the submitted label and clean_label as its reference on the same 50,000 images later audited, so the development and evaluation records were not disjoint. This could make the estimated efficiency optimistic. We therefore reran the mislabel-rate comparison without using clean labels to initialize predictions. Each policy learned whether a submitted label was correct only after selecting that image for audit. PAS retained its efficiency gain over the other baselines in this revised experiment (Table 1).

Q2) Flat or misspecified predictions and adaptive baselines

Could you add a run with flat or deliberately wrong priors, and a comparison with an adaptive finite-population method like Shekhar et al. (2023) at the same level and width?

Thank you; we added both analyses. We evaluate flat and deliberately wrong initial predictions, as well as systematic predictor misspecification through underconfidence, overconfidence, constant uncertainty, and reversed uncertainty rankings (Appendix D.8). PAS remains efficient under moderate misspecification, while severe overconfidence or incorrect rankings can eliminate its advantage.

For the two new adaptive baselines, see our response to W1 above.

Q3) Interpretation of the Bayesian model

In Section 6, is the Bayesian model a full joint posterior or an empirical-Bayes approximation with an updated pooled prior? This affects how to read the 95.5% credibility claim.

Thank you for the request for clarification. Section 6 uses an empirical-Bayes approximation, rather than a full joint posterior. We now describe the resulting 95.5% interval as an approximate credible interval with no distribution-free or anytime-valid guarantee.

Overall, we’re thankful for the strengths you found in our work and your helpful recommendations to improve its empirical rigor. The revised CIFAR-10N experiments, comparisons with established adaptive baselines, and sensitivity analyses across initializations strengthen the empirical case for our work.

Thanks for the thorough response. Adding the Shekhar et al. baselines and the two extra datasets answers my main concern, and I appreciate you confirming the CIFAR-10N overlap and rerunning without clean-label initialization. The misspecification study is useful too. I’d make the point that severe overconfidence or wrong rankings can remove the advantage clear in the main text, since that’s the practical limit of the method. Calling the Bayesian interval approximate also makes sense. I’m keeping my score at 3.

Thank you very much for the follow-up. We appreciate your kind words about the thoroughness of our response and are glad the new baselines and datasets addressed your main concern. We added a sentence in Section 4.1 (lines 305–306) clarifying that a consistently wrong predictor can, unsurprisingly, erase the efficiency advantage of PAS over simple random sampling, for example by reversing the priority ranking. We also point readers to Appendix D.8.3 for details on the experiments with severe overconfidence and incorrect rankings. If there is anything else you would like us to change or add, please let us know.

Reviewer_ofo4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a task-aware framework for auditing potentially mislabeled categorical datasets, adaptively prioritizing records whose uncertain corrections have the greatest influence on a downstream quantity while maintaining finite-sample, anytime-valid statistical guarantees.

The technical contributions combine prediction-corrected inverse-probability weighting with confidence sequences for linear scores, extend the framework to nonlinear scores such as Shannon diversity through linear envelopes and convex profiling, and provide a Bayesian alternative when distribution-free guarantees become too costly.

Experiments on a real Kenyan biodiversity dataset and CIFAR-10N demonstrate meaningful reductions in auditing cost, including 24%/11% fewer audits for mislabel-rate certification and 36% fewer audits for biodiversity-diversity certification compared with random sampling.

Critical strengths

1. Strong combination of adaptive efficiency and rigorous anytime-valid inference. The paper does not merely propose a heuristic for finding likely mislabels; it designs the audit around the downstream quantity of interest while preserving finite-sample, distribution-free confidence guarantees under both adaptive sampling and adaptive stopping. The prediction-corrected inverse-probability construction cleanly separates prediction quality from statistical validity: inaccurate predictions may reduce efficiency but do not invalidate the confidence sequence. This is consequential because it makes task-aware adaptive auditing usable in settings where statistical certification, rather than simply finding errors, is the primary objective.

2. Technically meaningful treatment of nonlinear downstream quantities with convincing empirical gains. The extension from linear scores to nonlinear functionals such as Shannon diversity is nontrivial: the paper exploits a linear-envelope representation, constructs separate upper and lower certificates, and turns the difficult lower bound into a certified convex optimization problem. Empirically, PAS reduces audits from 2,550 to 1,625 for the biodiversity-diversity target, while the CIFAR-10N experiments show substantial gains for worst-three precision and taxonomic risk. These results demonstrate that the approach extends beyond simple mislabel-rate estimation to task-specific quantities where the consequences of different label errors genuinely differ.

Critical weaknesses

1. The empirical evaluation is somewhat narrow relative to the claimed generality of the framework. The strongest distribution-free evidence comes from two datasets, the Kenyan biodiversity dataset and CIFAR-10N, with several downstream scores derived from them. While the experiments are thoughtful and include both linear and nonlinear targets, the effectiveness of priority-aware sampling depends explicitly on heterogeneous score-relevant uncertainty across strata. The reported gain already varies substantially, from 11% for CIFAR-10N mislabel-rate certification to 36% for biodiversity diversity, suggesting that practical benefit may be highly dataset- and target-dependent. Validation on additional domains and naturally heterogeneous audit problems would strengthen the claim of broad applicability.

2. The most statistically robust distribution-free procedures may still require substantial auditing, while the more efficient Bayesian alternative loses the paper’s strongest guarantee. For example, distribution-free certification of biodiversity diversity requires a median of 1,625 audits to obtain 5% relative accuracy. The Bayesian procedure reduces this dramatically to roughly 250 audits under priority-aware sampling, but its observed coverage is 93% rather than the nominal 95.5%, and random sampling achieves only 89%. This exposes an important practical tension: the rigorous method can remain expensive, while the practically cheaper alternative depends on modeling assumptions and is demonstrably somewhat overconfident on the primary real dataset.

Questions for authors
  1. Clarification: How sensitive are the efficiency gains of PAS to misspecification of the predictive distributions used to estimate stratum uncertainty? I understand that validity is preserved for any predictable predictor, which is an important strength, but the practical value depends on achieving meaningful sample savings. It would materially strengthen my assessment if the authors could quantify efficiency degradation under systematically misspecified or poorly calibrated predictors, especially during the early stages of an audit when few verified labels are available.

  2. Clarification: Can the authors provide additional guidance on when PAS should be expected to yield substantial gains over SRS based on observable pre-audit or early-audit properties of a dataset? The current results range from an 11% reduction in audits for CIFAR-10N mislabel-rate estimation to substantially larger improvements for nonlinear and biodiversity targets. A characterization or diagnostic connecting heterogeneity in score-relevant uncertainty to expected audit savings would make the practical contribution significantly stronger.

Suggestion for future work, not required for the current assessment: Extending the framework to imperfect or heterogeneous expert auditors, heterogeneous audit costs, and nonstationary correction patterns would be particularly valuable for real deployments.

1 comment

Thank you for your thoughtful review. We appreciate your recognition of our adaptive audit efficiency, anytime-valid inference, and extension to nonlinear scores.

W1) Breadth of evaluation

The empirical evaluation is somewhat narrow relative to the claimed generality of the framework.

Validation on additional domains and naturally heterogeneous audit problems would strengthen the claim of broad applicability.

We expanded the evaluation from two datasets to four by adding ReImageNet (30,908 image records) and CleanCoNLL (301,353 tokens from a named-entity recognition benchmark). On both datasets, we audit the overall mislabel rate and F1 to the same target precision of ±2 percentage points at 95.5% confidence.

For the mislabel rate, we added two anytime-valid baselines, Prop-MS and Prop-M + CV from Shekhar et al. (2023), using the authors’ implementation. PAS requires fewer audits than Prop-MS on all four datasets and fewer than Prop-M + CV on three of the four. PAS also reduces median audit counts relative to SRS on both new datasets (Table 1).

For nonlinear scores, we added PPR with projection, extending the prior–posterior-ratio method of Waudby-Smith and Ramdas (2020), and adapted the method from Manole and Ramdas (2023). PAS reduces the number of audits required to certify the nonlinear scores substantially when compared against these baselines (Table 8).

W2) Bayesian efficiency and coverage

The most statistically robust distribution-free procedures may still require substantial auditing, while the more efficient Bayesian alternative loses the paper’s strongest guarantee.

We agree that this is an important practical tradeoff. In the revised evaluation over 100 paired paths, the nominal 95.5% Bayesian interval covered the true diversity at stopping on 96 paths under both PAS and SRS, while whole-path coverage was 71 and 53 paths, respectively (Section 6, Table 2). We now emphasize that these intervals depend on modeling assumptions and are not anytime-valid. For high-stakes decisions requiring uniform coverage, we still recommend the distribution-free procedure. As we now discuss in Section 6, an open question is how closely a distribution-free, anytime-valid procedure can approach that efficiency.

Q1) Sensitivity to predictor misspecification

How sensitive are the efficiency gains of PAS to misspecification of the predictive distributions used to estimate stratum uncertainty?

It would materially strengthen my assessment if the authors could quantify efficiency degradation under systematically misspecified or poorly calibrated predictors, especially during the early stages of an audit when few verified labels are available.

Thank you for the suggestion. We added three sensitivity studies using 100 audit paths (Appendix D.8): sweeps over the exploration weight, initial predictions for both mislabel rate and diversity, and predictor misspecification through underconfidence, overconfidence, constant uncertainty, and reversed uncertainty.

PAS retained substantial gains (compared with SRS) across a broad range of exploration weights and moderate initialization errors. Its median mislabel-rate audit count varied only from 603 to 631 across initial correctness probabilities of 20–80%, and its diversity results remained similarly stable.

The misspecification study further characterizes the benefits of priority-awareness: mild prediction errors reduce but generally preserve the efficiency gain, while reversed rankings can make PAS slower than SRS. These experiments quantify the efficiency cost of poor predictions, but the anytime-valid guarantee remains unchanged.

Q2) When to expect gains from PAS

Can the authors provide additional guidance on when PAS should be expected to yield substantial gains over SRS based on observable pre-audit or early-audit properties of a dataset?

We expect substantial gains when a small share of records accounts for much of the uncertainty in the downstream score and the predictor reliably identifies the corresponding strata of these records.

A practical diagnostic is to estimate the standard deviation of a record’s score contribution within each stratum, using the initial predictions or the first few verified labels. SRS samples each stratum in proportion to its remaining size. The variance-based component of PAS weights that size by the estimated within-stratum standard deviation. When they differ substantially, PAS can direct more audits to strata where verifying a label is most informative about the score. Figure 8 in Appendix D.6 illustrates this point with a simple example in which only four of 20 strata contain records whose contribution to the score depends on the verified label.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Summary of Updates

We thank all the reviewers for their constructive feedback. We summarize the main changes below, with detailed responses in each reviewer’s thread.

Two new public datasets.

We evaluate and report performance on ReImageNet (image labels) and CleanCoNLL (named-entity labels in text) (Table 1). The full evaluation now covers four datasets.

Three new baselines and an ablation.

  • For the linear score (mislabel rate), we added two anytime-valid baselines, Prop-MS and Prop-M + CV from Shekhar et al. (2023), using the authors' code (Section 4.1).

  • For the five nonlinear scores, we added PPR with projection and adapted the method of Manole and Ramdas (2023). PPR with projection extends the PPR from Waudby-Smith and Ramdas (2020) baseline, which we already evaluated for the mislabel rate (Section 5.1).

  • We added an uncertainty-only ablation of PAS, which keeps the same confidence sequence, but prioritizes strata by the uncertainty of their predicted labels instead of their influence on the downstream score (Appendix D.5.1).

Further sensitivity and robustness experiments.

We added new studies of the SRS mixing weight, the initial predictions, and misspecified predictors, including an adversarial one (Appendix D.8). On the Kenya dataset, PAS had a median audit count that varied by less than 5% across a broad range of initial predictions for both the linear (mislabel rate) and nonlinear (Hill–Shannon diversity) scores. Misleading predictions can unsurprisingly reduce audit savings, while the confidence sequences retain their anytime-valid coverage guarantee.

Naming.

We named our framework ATLAS (Anytime-valid Task-aware Label Auditing and Sampling), with the algorithms being referred to as ATLAS-PAS and ATLAS-SRS. The new names distinguish them from the added baselines.

Presentation and related work.

We revised the paper throughout to present the ideas more clearly and make notation more consistent. We also added references suggested by the reviewers to the related work section.

Updated results.

  • Among anytime-valid methods, PAS requires the fewest audits for the mislabel rate on three of the four datasets (Table 1). On CleanCoNLL, Prop-M + CV has a lower median than PAS (233 vs. 281), but reaches the target on only 75 of 100 audit paths within 5000 audits, versus all 100 for PAS.

  • On all five nonlinear scores, PAS requires the fewest audits, and 6-38% fewer than the uncertainty-only ablation (Table 8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.