Stop Optimizing for the Middleman: Re-allocating Expert Time to Directly Improve Scientific Inference

Marginal improvements in classifier accuracy on scientific datasets can require significant annotation and optimization effort while yielding little improvement in the downstream quantities scientists ultimately seek to estimate. We study an alternative paradigm: allocate scarce human expertise according to its expected value for downstream scientific inference. Across scientific inference tasks, we show that this strategy can outperform spending the same labeling budget on classifier correction, and analyze differences between what data gets sampled in each case. We also explore classifier fine-tuning within this system and find that it can further boost downstream scientific inference, and that the strongest results come from selecting labels to verify with target-directed review and re-using these labels to update the classifier. Our results all emphasize the need for our field to move beyond classifier accuracy as the central objective when building expert verification systems in AI for science.