acceptodds
Under review as a conference paper at ICLR 2027

Stop Optimizing for the Middleman: Re-allocating Expert Time to Directly Improve Scientific Inference

Abstract

Marginal improvements in classifier accuracy on scientific datasets can require significant annotation and optimization effort while yielding little improvement in the downstream quantities scientists ultimately seek to estimate. We study an alternative paradigm: allocate scarce human expertise according to its expected value for downstream scientific inference. Across scientific inference tasks, we show that this strategy can outperform spending the same labeling budget on classifier correction, and analyze differences between what data gets sampled in each case. We also explore classifier fine-tuning within this system and find that it can further boost downstream scientific inference, and that the strongest results come from selecting labels to verify with target-directed review and re-using these labels to update the classifier. Our results all emphasize the need for our field to move beyond classifier accuracy as the central objective when building expert verification systems in AI for science.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
active inference, expert-in-the-loop, active learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cLuBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that when classifier outputs are only an intermediate step toward a scientific estimate, expert review should be allocated by its value for that estimate rather than for improving the classifier. The authors compare random review, classifier-directed review (predictive entropy, and BALD when labels are used for refitting) and target-directed review based on the expected information gain about the estimand. The comparison covers nine tasks built on bird acoustic surveys, Galaxy Zoo images and patent data, with errors measured against an oracle that uses all labels. Target-directed review reduces the error faster in eight of the nine tasks. The authors also find that it remains the most reliable choice when the reviewed labels are used to refit the classifier. In addition, the paper shows that classifiers with similar accuracy can lead to quite different downstream errors.

Critical strengths
  1. The question is important for AI for science and is framed clearly. Separating the two uses of an expert label, correcting the estimate directly versus improving the classifier, is a useful way to think about verification pipelines.

  2. The study is broad and generally careful. Each of the nine estimands is a quantity that scientists actually use in that domain, rather than a simple mean. Table 1 nicely shows how the selected data change with the target; for spatial clustering, for example, Target EIG picks galaxies close to those already reviewed. The VI approximation is validated against MCMC, and the limitations are discussed openly.

Critical weaknesses
  1. My main concern is that the comparison is one-sided, so the central finding is largely expected. Target EIG directly reduces uncertainty in the very quantity used for evaluation, while entropy, BALD and random review ignore it by design. The more interesting question is whether full EIG is needed. A simple target-aware score, such as classifier uncertainty weighted by the sensitivity of the estimate, used with the same Bayesian model would answer this. That matters because Target EIG is costly to compute. The claim that target-directed review is consistently the best strategy should also be softened, since it does not stay ahead on bar frequency.

  2. The evaluation only measures point error against an in-model oracle. This shows that target-directed review gets closer to the model's own full-label fit, not that it leads to better scientific conclusions. The paper does not examine three things: the calibration of the posterior intervals, robustness to misspecification, and how well labels chosen for one target serve the other targets. This matters because the selections are highly specialized (Table 1).

  3. Some design choices tilt the comparison. One patent review reveals only a single class label, although an expert reading a patent would assign all its classes at once. In the hybrid schedules, the BALD-selected labels are also used only to refit the classifier and never enter the scientific model.

Minors: Appendix G.1 has a heading but no text.

Questions for authors
  1. How does a simple target-aware score, such as uncertainty weighted by target sensitivity, compare with Target EIG when both use the same Bayesian model?
  2. Are the posterior intervals for the targets well calibrated along the review trajectories?
  3. How well do labels selected for one estimand work for the other estimands of the same dataset?
  4. How would the results change if one review revealed all labels of an item, such as all classes of a patent?
Reviewer_ubGoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission studies how expert effort should be allocated when classification is an intermediate step toward downstream scientific inference. Across 9 tasks spanning ecological surveys, galaxy morphology, and patent applications, the authors compare classifier-directed and target-directed review, with and without classifier refitting. Experimental results suggest that aggregate classification performance is an imperfect proxy for scientific utility, and expert verification should account for how individual observations contribute to the scientific target.

Critical strengths
  1. The submission addresses a practically relevant problem and clearly distinguishes predictive performance from downstream scientific inference.
  2. The experiments cover 3 domains and 9 tasks, and provide empirical analyses for target-directed review.
  3. The analysis of selected observations helps explain the observed improvements. It illustrates how the value of a reviewed label depends on the particular scientific target.
Critical weaknesses
  1. The experiments do not sufficiently separate the benefit of using downstream target information from the benefit of Target EIG specifically. The principal baselines, classifier entropy and BALD, do not take the downstream scientific target into account, whereas Target EIG explicitly uses this information and is evaluated on the corresponding target error. This comparison is appropriate for testing whether target-directed review is useful, but provides limited evidence for the advantage of EIG over other target-aware acquisition strategies.

  2. The mixed-schedule experiments confound acquisition strategy with how acquired labels are used. The submission states that BALD-selected labels are used only for classifier fitting, while EIG-selected labels are used only for direct scientific-model updates. In contrast, the all-EIG strategy uses every acquired label for both roles. So maybe weaker performance of the mixed schedules cannot be attributed solely to allocating reviews toward classifier-directed acquisition. A more controlled comparison would allow every strategy to reuse all acquired labels for both classifier refitting and downstream scientific inference, varying only the acquisition policy.

  3. Expert time is represented by the number of reviewed labels rather than measured annotation cost. Although the strategies acquire the same types of underlying labels, the selected observations may differ in review difficulty, required expertise, and opportunities to share annotation effort. The experiments also do not establish the tradeoff between fewer expert-verified labels and larger quantities of cheaper, potentially noisy labels. The results therefore demonstrate label efficiency under a fixed review-unit assumption, but do not yet establish superior allocation of expert time under realistic cost constraints.

Questions for authors

Please see above.

Reviewer_wEwkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a realistic problem, the dilemma between scarce expert resources and the need for high-quality data labeling. The authors compare three different approaches (1) random ; (2) classifier entropy based ; and (3) targeted directed review. The authors find that instead of optimizing against the classifier, it is better to optimize against the target directly.

Critical strengths
  1. it is a timely paper;
  2. a broad carefully controlled empircal demonstration across 9 tasks in 3 domains.
Critical weaknesses
  1. The baselines are weak, so the gain isn't attributed to its source. The classifier uncertainty baseline uses a frozen classifier that ignores everything the scientific model knows;
  2. Missing of error bars;
  3. The title is overclaimed.
Questions for authors
  1. If we add a baseline that uses the scientific model's own label uncertainty but ignores the target, how much of the target EIG's gain remains?
1 comment

The baselines are weak, so the gain isn't attributed to its source. The classifier uncertainty baseline uses a frozen classifier that ignores everything the scientific model knows;

If we add a baseline that uses the scientific model's own label uncertainty but ignores the target, how much of the target EIG's gain remains?

We added two target-directed baselines: (1) posterior predictive uncertainty, using the same Bayesian scientific model as Target EIG without taking into account the target, per your suggestion, and (2) sensitivity-weighted uncertainty, per Reviewer cLuB’s suggestion. Our updated comparison in Figure 3 includes these additional baselines. Our underlying claim remains unchanged: target-directed review reduces downstream error more efficiently than target-agnostic review (though there may be variation between the target-directed strategies). We have updated our discussion in Section 4.2 to reflect these changes.

Missing of error bars;

We have now added the missing error bars for the abundance results in Figure 3 and Table 5. We will furthermore add error bars to Figure 4 in the camera-ready version. Please let us know if we missed error bars on any other results, and we would be happy to make the revision.

The title is overclaimed.

Our central claim is that when classification is an intermediate step toward scientific inference, optimizing classifier accuracy alone may not be the most effective use of limited expert verification effort. In Section 4.3, we clarify that our argument is not that classifier quality is unimportant, and our experiments demonstrate that stronger classifiers and classifier refitting can improve downstream inference. However, we believe that our current title emphasizes the underlying claim that the way labels are acquired should be aligned with the downstream scientific target, and respectfully prefer to keep it.

Comments on the paper

2 comments

We thank all three reviewers for appreciating the practical implications of this work and the breadth of its evaluation and analysis, and for their thoughtful and constructive feedback. In response to concerns about the strength of our baselines (wEwk, ubGo, cLuB), we have added posterior predictive uncertainty and sensitivity-weighted uncertainty as additional baselines in Figure 3 to better isolate the effects of target-aware acquisition from target-agnostic strategies more generally. Reviewer ubGo raised concerns about confounding in our refitting experiments. To address this, we added a comparison where all reviewed labels, no matter if they resulted from BALD or EIG selection, update both the classifier and scientific model (see Appendix G.6). We have also clarified our assumptions (see “Review protocol” under Section 3.3) and softened our claims and contributions throughout the text (see Contribution 3 in the Intro, Section 4.2, and Conclusion), per Reviewers wEwk and cLuB’s comments. Within the revised PDF, additions are highlighted in dark green, deletions are struck through and colored red, and updated text is colored dark blue.

Dear Reviewers,

Thank you again for your thoughtful comments and feedback. We believe that your comments and suggested additional experiments helped meaningfully improve the paper. We wanted to follow up on our responses below and see if there are any further questions or concerns we could help address before the review period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.