- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on the setting where the “ground truth” data (for a finite dataset) used for downstream inference are noisy, and need to be reviewed with limited expert effort. The proposed solution uses a confusion matrix to represent label corrections and prioritizes verification effort on data that most narrow confidence intervals for the downstream task. The authors demonstrate the effects of this approach on using wildlife data for biodiversity credit decision-making.
Critical strengths
- The problem setting of active review is well motivated, and the proposed method is evaluated on a real-world application where it is clearly relevant.
- The authors provide strong intuition for assumption-free anytime-valid audits.
- The paper ends with a discussion of limitations and opportunities for future work.
Critical weaknesses
- Several citations are missing from the Related Works. Consider adding citations for post-prediction inference [1], and prediction-powered inference (PPI) [2].
- While I am convinced that this problem setting is important, this argument/perspective is not necessarily novel in itself. This paper [3] is also a relevant example of a paper that does active review of noisy ML outputs for a specific downstream inference target. The language should reflect the fact that other workshave considered a similar setting.
- The overall framework echoes this method for active model selection [4]. Consider adding a citation to the Related Works, and discussing how they are complimentary.
- Include standard, non-random-based baselines (a downstream entropy-based sampling, for example).
- There are several key missing details about the methodology. The datasets are not cited and the downstream goals optimized for are not sufficiently described in the main text.
- Report results over seeds in the tables for the means and coverages.
- The discussion of results is dense with numerical results, making it hard to parse the high-level takeaways. The results are not sufficiently discussed in the main text (i.e. the confusion matrix in Figure 2 is not directly mentioned).
[1] Wang, S., McCormick, T. H., & Leek, J. T. (2020). Post-prediction inference. BioRxiv, 2020-01. [2] Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., & Zrnic, T. (2023). Prediction-powered inference. Science, 382(6671), 669-674. [3] Haucke, T., Harrell, L., Kay, J., Clapp, M., & Beery, S. (2026). Targeted Review for AI-Assisted Biodiversity Surveys: Active Continuous-Score Occupancy Modeling. arXiv preprint arXiv:2609.25657. [4] Kay, J., Van Horn, G., Maji, S., Sheldon, D., & Beery, S. (2025, October). Consensus-driven active model selection. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 4594-4604). IEEE.
Questions for authors
- What differentiates this method from previous active review literature?
- What determines when an expert should stop reviewing in practice? Is the only stopping condition a constraint on the confidence or relative accuracy? What determines the stopping point for the figures and the tables?
4 comments
Q2) Practical stopping and reported audit counts
What determines when an expert should stop reviewing in practice?
The auditor can stop when the interval is sufficiently precise for the decision or when the review budget is exhausted. If the budget runs out first, the current interval and achieved precision are reported without claiming that the target was met. For the figures and tables, we record the first evaluated checkpoint at which the 95.5% interval meets an arbitrary prespecified target: for example, a half-width of two percentage points for mislabel rate, or 5% of the interval midpoint for Hill–Shannon diversity. Domain-experts would select these thresholds in practice.
Thank you again for highlighting the practical relevance of our work and suggesting connections to the active-review literature. Your feedback helped us better present our contribution, strengthen the comparisons, and make the experimental goals and results clearer.
Thank you for your thoughtful and detailed response. I have reviewed the updated submission, and I believe that the edits meaningfully improved it. In particular, the revised Related Works section definitely helps clarify the contribution of this work, and the uncertainty-only ablation and additional datasets strengthen the comparisons made in the paper. It is very clear that task-aware review is more efficient than standard uncertainty-based review. Just one more question: since improvements over uncertainty-driven review is variable, it would be interesting to discuss when using more complex task-aware selection is actually justified. This is more a point of further clarification. I am increasing my score from a 2 (weak reject) to a 3 (weak accept).
Thank you very much for your encouraging follow-up. We’re glad the revisions addressed your concerns.
To answer your question: task-aware selection is most useful when labels with similar uncertainty have different consequences on the downstream score. For example, suppose two camera locations have similarly uncertain predictions: one mainly confuses cattle with sheep while the other confuses cattle with lions. Sheep and cattle are both common in the dataset, but lions are rare, and therefore more influential on the downstream diversity score. Resolving the second location’s labels therefore matters more, so task-aware selection can prioritize it even when uncertainty-only selection treats the locations similarly. This illustrates the differences in score-relevant uncertainty discussed in our limitations (lines 532–534).
We appreciate your careful reassessment and increasing your score from 2 to 3. There may be a delay in OpenReview, but so far we do not see the score update yet.
Thank you for your thoughtful review. We appreciate your recognition of the problem’s practical relevance, the intuition behind our assumption-free anytime-valid audits, and the discussion of limitations and future directions.
W1–W3 and Q1) Related work and contribution
Thank you for the related work suggestions. We revised Section 7 to cite and discuss Wang et al. (2020), Angelopoulos et al. (2023), Haucke et al. (2026), and Kay et al. (2025), acknowledging prior work on both prediction-assisted inference and active review. Our framework additionally certifies the resulting quantity with an anytime-valid uncertainty guarantee. Our contribution is to combine task-aware adaptive review with finite-sample, distribution-free confidence sequences that remain valid under both adaptive sampling and adaptive stopping, including for nonlinear downstream quantities. For a given score, our PAS uses Bayesian predictions of label corrections to estimate how much each record’s contribution varies within its stratum. These estimates guide sampling toward the variance-minimizing allocation characterized in Proposition 3. Shekhar et al. (2023) provides a direct comparison for linear scores because their method supports anytime-valid inference under weighted sampling without replacement. For comparative results against a collection of new baselines, see W4 below.
W4) Non-random baselines
Thank you for the suggestion. For mislabel rate, we evaluated Prop-MS and Prop-M + CV from Shekhar et al. (2023) using the authors’ released code. Table 1 reports comparisons across four datasets, including the newly added ReImageNet and CleanCoNLL. PAS required significantly fewer audits than Prop-MS on all four datasets and Prop-M + CV on three of four (Table 1).
We also certify nonlinear scores through a linear-envelope representation of entropy and a linear-ratio representation of F1. We added PPR with projection, extending the prior–posterior-ratio method of Waudby-Smith and Ramdas (2020), and adapted the method from Manole and Ramdas (2023). PAS reduces the number of audits required to certify the nonlinear scores substantially when compared against these baselines (Table 8).
To test whether audit selection improves when it accounts for how each possible label correction affects the score, we compared PAS with an adaptive policy based on label uncertainty alone. Across the five nonlinear scores considered, PAS reduced the median audit count by 6%–38% relative to this baseline (Table 8).
W5) Dataset and downstream-task descriptions
Section 4.1 now cites CIFAR-10N, ReImageNet, and CleanCoNLL; Appendix D.1 describes the Kenyan dataset and its source. We also release the Kenya confusion matrix with species names. In our linear-score experiments, the task is to certify mislabel rate to within two percentage points at a nominal 95.5% confidence level using as few audits as possible (Section 4.1). Section 5.1 now describes each nonlinear score and its precision target.
W6) Variation across audit paths and coverage
Thank you for the suggestion. All tables now report the median audit count and the 25th and 75th percentiles across 100 audit paths. Figure 10 shows how often the intervals covered the true Kenyan mislabel rate and diversity score through the full audit path while varying initial predictions. Table 9 reports whole-path coverage for all five nonlinear tasks. For the Bayesian procedure, Table 2 reports coverage both at stopping and at every checkpoint through stopping. The supplementary material includes the random seeds used in the experiments.
W7) Clearer results discussion
We rewrote Sections 4.1 and 5.1 to emphasize performance trends across baselines and direct readers to the appendix for detailed comparisons. Figure 2’s confusion matrix displays submitted-to-verified label patterns in the Kenyan input dataset, from which all methods sample. We clarified this role in the text (Section 2, Section 4.1).