Same Budget, Different Learner: Schedule Gap in Active Learning

Active learning (AL) aims to reduce labeling effort by selecting examples for annotation. AL methods are commonly compared by accuracy at the same annotation budget. However, many small acquisition batches or a few large ones can reach the same budget with different model updates and potentially different accuracies. We study this overlooked schedule dependence by defining the schedule gap, which measures the average accuracy difference between acquisition schedules at shared cumulative budgets. Across ten methods and five natural and medical image classification datasets, the mean schedule gap ranges from 1.42% to 10.25%, and methods with low schedule sensitivity do not necessarily achieve the strongest predictive performance. Motivated by this observation, we introduce ALPS, a scheduler that distributes each acquisition batch across representativeness, coverage, and uncertainty using the estimated learner state. ALPS achieves the highest average predictive performance while maintaining a mean schedule gap of 1.78%, with its largest gains on the medical imaging datasets. Our results show that cumulative budget reflects acquired supervision, but does not fully characterize the learner state or which acquisition strategy is most useful next.