- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper points out that the batch schedule of active learning matters and suggests an algorithm (ALPS) which creates an adaptive mixture of uncertainty, coverage, and representativeness samples. This algorithm is applied to several datasets showing that it's competitive with other deep active learning algorithms yielding modestly better average performance with modestly better consistency.
Critical strengths
It is plausible that an adaptive strategy can yield a more robust active learning algorithm, and this paper creates a partially adaptive active learning algorithm.
Critical weaknesses
Batch schedules have been studied extensively before. See for example: Amin, Cortes, DeSalvo, Rostamizadeh, "Understanding the effects of batching in online active learning," AISTATS 2020 Chen & Krause, "Near-optimal batch mode active learning and adaptive submodular optimization," ICML 2013 Golovin & Krause, "Adaptive submodularity," JAIR 2011 There's also more theoretical work showing that on first principles you would actually like an adaptive batch size with few rounds of interaction. See for example: Esfandiari, Karbasi & Mirrokni, "Adaptivity in adaptive submodularity" (COLT 2021) and even: Balcan, Beygelzimer & Langford (A², 2006)
The claimed improvements over other algorithms are modest with a modest evaluation. Typically, I like to see either a large improvement with a modest evaluation or a modest improvement with a large evaluation. In the current state, it seems hard to be confident that ALPS is actually better in practice than other deep AL algorithms, since adding a dataset to the evaluation might change the rankings.
The experimental result reporting seems to be moderately inconsistent. Figure uses 1000&500 while table 3 uses 1000 and 100.
Questions for authors
The estimates formed for state tracking require a validation label set which appears to not be accounted for in the experimental methodology?
5 comments
Validation supervision (Q1)
We agree that ALPS uses validation labels beyond the checkpoint selection performed by all methods (App. C.2). To isolate this additional supervision, we evaluate ALPS-geo, which estimates the learner state using only geometric information, while retaining the use of the validation set for checkpoint selection as all other methods.
Across the five evaluated datasets, removing the predictive deficit (i.e., removing additional validation supervision) changes AULC by at most 0.47 points on four datasets, with a larger reduction of 1.73 on BRISC2025. ALPS-geo maintains a mean schedule gap of 1.84%, compared with 1.78% for ALPS. Thus, ALPS remains competitive without using validation labels to guide acquisition, although estimating the predictive deficit provides additional benefits, especially on BRISC2025. Full results are provided in App. D.5, Tab. 11.
My personal view is that the paper would be strengthened by discarding ALPS and only talking about ALPS-geo (at a semantic, not syntactic level) since that would reduce the complexity of the results and ALPS-geo is likely what people would want to use in practice.
I also don't yet find the experimental results compelling, because looking at the dataset to dataset variation it appears that the addition or subtraction of one dataset to your benchmark could scramble the order of the top results in figure 3 left.
We thank the reviewer for the constructive suggestions and for highlighting relevant theoretical work on batching and adaptivity in active learning. In response, we updated the manuscript to clarify our contributions relative to prior work (Sec. 2), evaluated TCM across five datasets and extended annotation budgets (Sec. 5.2, App. D.8), investigated additional schedules and validation supervision (Apps. D.5 and D.6), and updated the allocation visualization (Fig. 4, Sec. 5.3). These additions strengthen our original conclusions while clarifying their scope.
Prior work on batching and adaptivity (W1)
We agree that batching have been extensively studied, including through adaptive submodularity (Golovin & Krause, 2011), batch-mode selection (Chen & Krause, 2013), online active learning (Amin et al., 2020), and limited-round adaptive optimization (Esfandiari et al., 2021). We expanded the related work to include all five papers highlighted by the reviewer (Sec. 2). These works provide important theoretical results, but our setting and objective differ in two respects.
First, we study schedule sensitivity in deep pool-based AL, where different acquisition schedules can produce different learning trajectories and accuracies at the same cumulative budget. Rather than establishing another theoretical guarantee for batching, we quantify this empirical disagreement across established methods and datasets through the schedule gap (Sec. 3.1). Additional seed controls and mean-curve analyses further clarify its interpretation (App. D.7, Tab. 14).
Second, ALPS does not optimize the batch size or number of acquisition rounds. It addresses settings where the schedule is externally specified and adapts the composition of each batch according to the estimated learner state (Secs. 3.2 and 4).
Thus, our contributions are the empirical characterization of schedule dependent accuracy disagreement in deep AL and a learner-state-conditioned scheduler for externally specified acquisition batches, rather than introducing batching or adaptive selection as new concepts.
Predictive performance and evaluation scope (W2)
We agree that ALPS does not achieve the highest AULC on every dataset, particularly compared with TypiClust on natural images under low annotation budgets. Our objective is to maintain strong predictive performance while limiting sensitivity to the acquisition schedule.
Across five datasets, ALPS achieves the smallest mean AULC deficit (0.82%) with a mean schedule gap of 1.78%, compared with 1.24% and 1.86% for the newly evaluated TCM baseline (Sec. 5.2, Tab. 9). Its strongest advantages occur on the medical datasets, while other methods perform better in some natural image settings.
We additionally evaluated three acquisition schedules on two datasets (App. D.6, Tab. 13) and extended the annotation budget ranges. Notably, ALPS outperforms TypiClust on all four extended budget datasets, including CIFAR-10 and CIFAR-100, where TypiClust leads in the original low-budget evaluation (App. D.8, Tab. 16). Other methods, including TCM, remain stronger in some settings.
These results reinforce that method rankings depend on the dataset and evaluation regime. Therefore, our conclusions concern the favorable performance-consistency trade-off of ALPS, rather than universal superiority.
Acquisition schedules in Fig. 4 (W3)
We thank the reviewer for identifying this discrepancy. We have updated Fig. 4 to use the main evaluation batch sizes for CIFAR-100, SVHN, ISIC2019, and BRISC2025. For CIFAR-10, we retain a separate illustrative setting to show how ALPS allocation evolves over a longer annotation-budget range. We have explicitly clarified the corresponding batch sizes and their purpose in Sec. 5.3 and the figure caption. Importantly, Fig. 4 is intended to illustrate allocation dynamics, whereas all quantitative AULC and schedule-gap comparisons follow the experimental protocol specified in Tab. 3. Therefore, the illustrative setting does not affect the quantitative results or conclusions.