- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that when classifier outputs are only an intermediate step toward a scientific estimate, expert review should be allocated by its value for that estimate rather than for improving the classifier. The authors compare random review, classifier-directed review (predictive entropy, and BALD when labels are used for refitting) and target-directed review based on the expected information gain about the estimand. The comparison covers nine tasks built on bird acoustic surveys, Galaxy Zoo images and patent data, with errors measured against an oracle that uses all labels. Target-directed review reduces the error faster in eight of the nine tasks. The authors also find that it remains the most reliable choice when the reviewed labels are used to refit the classifier. In addition, the paper shows that classifiers with similar accuracy can lead to quite different downstream errors.
Critical strengths
-
The question is important for AI for science and is framed clearly. Separating the two uses of an expert label, correcting the estimate directly versus improving the classifier, is a useful way to think about verification pipelines.
-
The study is broad and generally careful. Each of the nine estimands is a quantity that scientists actually use in that domain, rather than a simple mean. Table 1 nicely shows how the selected data change with the target; for spatial clustering, for example, Target EIG picks galaxies close to those already reviewed. The VI approximation is validated against MCMC, and the limitations are discussed openly.
Critical weaknesses
-
My main concern is that the comparison is one-sided, so the central finding is largely expected. Target EIG directly reduces uncertainty in the very quantity used for evaluation, while entropy, BALD and random review ignore it by design. The more interesting question is whether full EIG is needed. A simple target-aware score, such as classifier uncertainty weighted by the sensitivity of the estimate, used with the same Bayesian model would answer this. That matters because Target EIG is costly to compute. The claim that target-directed review is consistently the best strategy should also be softened, since it does not stay ahead on bar frequency.
-
The evaluation only measures point error against an in-model oracle. This shows that target-directed review gets closer to the model's own full-label fit, not that it leads to better scientific conclusions. The paper does not examine three things: the calibration of the posterior intervals, robustness to misspecification, and how well labels chosen for one target serve the other targets. This matters because the selections are highly specialized (Table 1).
-
Some design choices tilt the comparison. One patent review reveals only a single class label, although an expert reading a patent would assign all its classes at once. In the hybrid schedules, the BALD-selected labels are also used only to refit the classifier and never enter the scientific model.
Minors: Appendix G.1 has a heading but no text.
Questions for authors
- How does a simple target-aware score, such as uncertainty weighted by target sensitivity, compare with Target EIG when both use the same Bayesian model?
- Are the posterior intervals for the targets well calibrated along the review trajectories?
- How well do labels selected for one estimand work for the other estimands of the same dataset?
- How would the results change if one review revealed all labels of an item, such as all classes of a patent?
We added two target-directed baselines: (1) posterior predictive uncertainty, using the same Bayesian scientific model as Target EIG without taking into account the target, per your suggestion, and (2) sensitivity-weighted uncertainty, per Reviewer cLuB’s suggestion. Our updated comparison in Figure 3 includes these additional baselines. Our underlying claim remains unchanged: target-directed review reduces downstream error more efficiently than target-agnostic review (though there may be variation between the target-directed strategies). We have updated our discussion in Section 4.2 to reflect these changes.
We have now added the missing error bars for the abundance results in Figure 3 and Table 5. We will furthermore add error bars to Figure 4 in the camera-ready version. Please let us know if we missed error bars on any other results, and we would be happy to make the revision.
Our central claim is that when classification is an intermediate step toward scientific inference, optimizing classifier accuracy alone may not be the most effective use of limited expert verification effort. In Section 4.3, we clarify that our argument is not that classifier quality is unimportant, and our experiments demonstrate that stronger classifiers and classifier refitting can improve downstream inference. However, we believe that our current title emphasizes the underlying claim that the way labels are acquired should be aligned with the downstream scientific target, and respectfully prefer to keep it.