Direct Optimization of Selective Classifier for Auto-Labeling via Exact Reformulation

With the increasing demand for large-scale labeled data, auto-labeling has emerged as an effective technique for annotating unlabeled data with minimal human intervention. The goal of auto-labeling is to achieve a coverage as high as possible on the unlabeled data while keeping the error rate below a user-specified value. Threshold-based auto-labeling (TBAL) is a multi-round framework that leverages a classifier to selectively label samples based on whether the confidence scores exceed an estimated threshold. However, standard classifier training does not take into account the error-coverage tradeoff in auto-labeling and may yield confidence scores that poorly separate correct from incorrect predictions. Furthermore, directly optimizing this objective is challenging because both coverage and error involve indicator functions with zero gradients almost everywhere, while replacing them with smooth surrogates introduces an approximation gap that undermines optimality. In this paper, we propose AL-ERO (Auto-Labeling via Exact Reformulation and Optimization), which trains a selective classifier by jointly optimizing classification and selection. AL-ERO uses an exact reformulation of the selection indicator, thereby eliminating the approximation gap introduced by smooth surrogates. Experiments on four datasets covering image and text modalities show that AL-ERO achieves improved auto-labeling performance.