acceptodds
Under review as a conference paper at ICLR 2027

Same Budget, Different Learner: Schedule Gap in Active Learning

17 views

Abstract

Active learning (AL) aims to reduce labeling effort by selecting examples for annotation. AL methods are commonly compared by accuracy at the same annotation budget. However, many small acquisition batches or a few large ones can reach the same budget with different model updates and potentially different accuracies. We study this overlooked schedule dependence by defining the schedule gap, which measures the average accuracy difference between acquisition schedules at shared cumulative budgets. Across ten methods and five natural and medical image classification datasets, the mean schedule gap ranges from 1.42% to 10.25%, and methods with low schedule sensitivity do not necessarily achieve the strongest predictive performance. Motivated by this observation, we introduce ALPS, a scheduler that distributes each acquisition batch across representativeness, coverage, and uncertainty using the estimated learner state. ALPS achieves the highest average predictive performance while maintaining a mean schedule gap of 1.78%, with its largest gains on the medical imaging datasets. Our results show that cumulative budget reflects acquired supervision, but does not fully characterize the learner state or which acquisition strategy is most useful next.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Active Learning, Schedule Gap, Acquisition Schedule, Data-Efficient Learning, Sample Efficiency, Adaptive Acquisition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gngQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper points out that the batch schedule of active learning matters and suggests an algorithm (ALPS) which creates an adaptive mixture of uncertainty, coverage, and representativeness samples. This algorithm is applied to several datasets showing that it's competitive with other deep active learning algorithms yielding modestly better average performance with modestly better consistency.

Critical strengths

It is plausible that an adaptive strategy can yield a more robust active learning algorithm, and this paper creates a partially adaptive active learning algorithm.

Critical weaknesses

Batch schedules have been studied extensively before. See for example: Amin, Cortes, DeSalvo, Rostamizadeh, "Understanding the effects of batching in online active learning," AISTATS 2020 Chen & Krause, "Near-optimal batch mode active learning and adaptive submodular optimization," ICML 2013 Golovin & Krause, "Adaptive submodularity," JAIR 2011 There's also more theoretical work showing that on first principles you would actually like an adaptive batch size with few rounds of interaction. See for example: Esfandiari, Karbasi & Mirrokni, "Adaptivity in adaptive submodularity" (COLT 2021) and even: Balcan, Beygelzimer & Langford (A², 2006)

The claimed improvements over other algorithms are modest with a modest evaluation. Typically, I like to see either a large improvement with a modest evaluation or a modest improvement with a large evaluation. In the current state, it seems hard to be confident that ALPS is actually better in practice than other deep AL algorithms, since adding a dataset to the evaluation might change the rankings.

The experimental result reporting seems to be moderately inconsistent. Figure uses 1000&500 while table 3 uses 1000 and 100.

Questions for authors

The estimates formed for state tracking require a validation label set which appears to not be accounted for in the experimental methodology?

5 comments

We thank the reviewer for the constructive suggestions and for highlighting relevant theoretical work on batching and adaptivity in active learning. In response, we updated the manuscript to clarify our contributions relative to prior work (Sec. 2), evaluated TCM across five datasets and extended annotation budgets (Sec. 5.2, App. D.8), investigated additional schedules and validation supervision (Apps. D.5 and D.6), and updated the allocation visualization (Fig. 4, Sec. 5.3). These additions strengthen our original conclusions while clarifying their scope.

Prior work on batching and adaptivity (W1)

We agree that batching have been extensively studied, including through adaptive submodularity (Golovin & Krause, 2011), batch-mode selection (Chen & Krause, 2013), online active learning (Amin et al., 2020), and limited-round adaptive optimization (Esfandiari et al., 2021). We expanded the related work to include all five papers highlighted by the reviewer (Sec. 2). These works provide important theoretical results, but our setting and objective differ in two respects.

First, we study schedule sensitivity in deep pool-based AL, where different acquisition schedules can produce different learning trajectories and accuracies at the same cumulative budget. Rather than establishing another theoretical guarantee for batching, we quantify this empirical disagreement across established methods and datasets through the schedule gap (Sec. 3.1). Additional seed controls and mean-curve analyses further clarify its interpretation (App. D.7, Tab. 14).

Second, ALPS does not optimize the batch size or number of acquisition rounds. It addresses settings where the schedule is externally specified and adapts the composition of each batch according to the estimated learner state (Secs. 3.2 and 4).

Thus, our contributions are the empirical characterization of schedule dependent accuracy disagreement in deep AL and a learner-state-conditioned scheduler for externally specified acquisition batches, rather than introducing batching or adaptive selection as new concepts.

Predictive performance and evaluation scope (W2)

We agree that ALPS does not achieve the highest AULC on every dataset, particularly compared with TypiClust on natural images under low annotation budgets. Our objective is to maintain strong predictive performance while limiting sensitivity to the acquisition schedule.

Across five datasets, ALPS achieves the smallest mean AULC deficit (0.82%) with a mean schedule gap of 1.78%, compared with 1.24% and 1.86% for the newly evaluated TCM baseline (Sec. 5.2, Tab. 9). Its strongest advantages occur on the medical datasets, while other methods perform better in some natural image settings.

We additionally evaluated three acquisition schedules on two datasets (App. D.6, Tab. 13) and extended the annotation budget ranges. Notably, ALPS outperforms TypiClust on all four extended budget datasets, including CIFAR-10 and CIFAR-100, where TypiClust leads in the original low-budget evaluation (App. D.8, Tab. 16). Other methods, including TCM, remain stronger in some settings.

These results reinforce that method rankings depend on the dataset and evaluation regime. Therefore, our conclusions concern the favorable performance-consistency trade-off of ALPS, rather than universal superiority.

Acquisition schedules in Fig. 4 (W3)

We thank the reviewer for identifying this discrepancy. We have updated Fig. 4 to use the main evaluation batch sizes for CIFAR-100, SVHN, ISIC2019, and BRISC2025. For CIFAR-10, we retain a separate illustrative setting to show how ALPS allocation evolves over a longer annotation-budget range. We have explicitly clarified the corresponding batch sizes and their purpose in Sec. 5.3 and the figure caption. Importantly, Fig. 4 is intended to illustrate allocation dynamics, whereas all quantitative AULC and schedule-gap comparisons follow the experimental protocol specified in Tab. 3. Therefore, the illustrative setting does not affect the quantitative results or conclusions.

Validation supervision (Q1)

We agree that ALPS uses validation labels beyond the checkpoint selection performed by all methods (App. C.2). To isolate this additional supervision, we evaluate ALPS-geo, which estimates the learner state using only geometric information, while retaining the use of the validation set for checkpoint selection as all other methods.

Across the five evaluated datasets, removing the predictive deficit (i.e., removing additional validation supervision) changes AULC by at most 0.47 points on four datasets, with a larger reduction of 1.73 on BRISC2025. ALPS-geo maintains a mean schedule gap of 1.84%, compared with 1.78% for ALPS. Thus, ALPS remains competitive without using validation labels to guide acquisition, although estimating the predictive deficit provides additional benefits, especially on BRISC2025. Full results are provided in App. D.5, Tab. 11.

My personal view is that the paper would be strengthened by discarding ALPS and only talking about ALPS-geo (at a semantic, not syntactic level) since that would reduce the complexity of the results and ALPS-geo is likely what people would want to use in practice.

I also don't yet find the experimental results compelling, because looking at the dataset to dataset variation it appears that the addition or subtraction of one dataset to your benchmark could scramble the order of the top results in figure 3 left.

Reviewer_wycDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how acquisition schedules affect active learning performance under the same annotation budget. The authors introduce the schedule gap to measure performance differences between schedules, and evaluate existing methods on natural and medical image datasets. They also propose ALPS, which estimates the learner state from feature discrepancy and validation error, and adaptively allocates samples across three acquisition strategies. ALPS shows the lowest average AULC deficit, mainly from the medical datasets.

Critical strengths
  1. Schedule as an evaluation axis.
  • The paper shows that the same annotation budget can give different accuracies depending on the acquisition schedule.
  • The evaluation across many methods and datasets gives useful insights for AL benchmarking.
  1. Simple method with strong medical results.
  • ALPS combines existing acquisition methods through adaptive allocation, without modifying their mechanisms.
  • It achieves the highest AULC with low schedule gaps on both medical datasets.
Critical weaknesses
  1. The reliability of the schedule gap could be further validated.
  • Stochastic variability: Since the metric uses absolute differences, it may give positive gaps even without a real schedule effect. Random sampling is independent of the learner state, but it still shows a mean gap of 1.42%. This may partly reflect stochastic variability rather than genuine schedule dependence. It would be helpful to include a same-schedule control to distinguish stochastic variability from schedule-dependent effects.
  • Limited evaluation: Only one schedule pair is tested per dataset. Also, the upper value of 10.25% comes from ProbCover, mainly due to its large gaps on the medical datasets (23.80%). Without ProbCover, the maximum mean gap is 4.26%.
  • Missing ablations: Table 1 reports only AULClog under a single schedule, with budget ranges different from the main experiments. Therefore, it is not yet clear whether ALPS components improve schedule consistency.
  1. The advantage of state-conditioned allocation seems not fully established.
  • Validation supervision: All methods use the same validation set for checkpoint selection, but ALPS additionally uses its labels to guide acquisition. This may limit its applicability when labeled validation data is scarce.
  • Missing baselines: Closely related adaptive methods, especially TCM (Doucet et al., 2024), are not compared. SelectAL and ALBL are also not included in the experiments. A comparison with these methods would make the contribution of ALPS clearer.
  • Concentrated gains: ALPS achieves the highest AULC only on the two medical datasets, while TypiClust performs better on all three natural image datasets and on ImageNet-200.
  • Limited evidence: Some ablations show clear performance degradation, but several differences are small relative to the reported variability. In addition, the allocation mapping is compared with only one alternative, and a sensitivity analysis for τ, β, and the number of midpoints would be helpful.
Questions for authors

Clarification questions

  1. Schedule-gap reliability: What gap is observed between independent runs with the same schedule? Also, is the classifier retrained from scratch at every round?
  2. Schedule-gap ablations: Can the authors report schedule gaps for the key ablation variants under the same settings as the main experiments?
  3. Validation supervision: How does ALPS perform with a much smaller validation set, or without using validation labels for acquisition?
  4. Adaptive baselines: How does ALPS compare with TCM, SelectAL, and ALBL under the same experimental protocol?

Suggestions

  1. Please evaluate additional acquisition schedules, and report signed accuracy differences together with the absolute schedule gap.
  2. Please provide a clearer analysis of the trade-off between AULC and schedule gap, and consider balanced accuracy or macro-F1 for the class-imbalanced ISIC2019 dataset.
6 comments

We thank the reviewer for the constructive suggestions. In response to the main concerns, we conducted additional experiments and updated the paper accordingly. Specifically, we (1) analyzed seed variability and signed schedule differences (App. D.7, Tabs. 14 and 15), (2) extended the evaluation to three schedules, including schedule gap comparisons for main ALPS variants (App. D.6, Tab. 13), (3) investigated the impact of validation supervision (App. D.5, Tabs. 11 and 12), and (4) compared ALPS against TCM across all five datasets (Sec. 5.2, Tabs. 4 to 9). We additionally evaluated macro-F1, extended annotation budgets, and controller sensitivity (App. D.8, Tabs. 16 to 18). These results strengthen our original conclusions while clarifying their scope, as detailed below.

Schedule gap reliability (W1, Q1, S1)

We agree that stochastic variability can produce positive schedule gaps even without systematic schedule dependent differences. Therefore, we introduce a within-schedule seed control and measure differences between seed-averaged learning curves (App. D.7, Tab. 14).

On BRISC2025, the schedule gap of ALPS is comparable to its seed variability (0.83 vs 0.89), while Entropy exhibits a much larger gap (6.63 vs. 1.53). After averaging learning curves across seeds, Entropy still shows substantial disagreement (6.54), compared with 0.49 for ALPS. Thus, schedule sensitivity differs across methods and cannot be characterized by stochastic variability alone, although these controls do not fully isolate a causal schedule effect.

Additionally, we report signed schedule differences (App. D.7, Tab. 15), showing that some methods consistently favor one schedule over another.

We also clarified that classifiers are reinitialized and retrained from scratch after every acquisition round (App. C.1), and that the mean schedule gap of 10.25% of ProbCover was obtained using a fixed radius without dataset-specific calibration. Therefore, we report its results for completeness but exclude them from our main comparison and correctly acknowledge that the maximum mean schedule gap of the evaluated methods is 4.26% (Abstract and Sec. 5.2).

Additional schedules and component consistency (W1, Q2, S1)

We introduce a third acquisition schedule on BRISC2025 (b=20,100,200b=20,100,200) and CIFAR-100 (b=100,500,1000b=100,500,1000), covering medical and natural and medical image classification. We evaluate all three schedule pairs for ALPS and its main variants (App. D.6, Tab. 13).

ALPS achieves smaller schedule gaps than BADGE and TypiClust (the baselines with the highest AULC on the selected datasets) across all six pairwise comparisons, with higher AULC on BRISC2025 and slightly lower on CIFAR-100. Importantly, removing within-batch integration increases the schedule gap on all six schedule pairs. On BRISC2025, the fine/coarse schedule gap increases from 0.83% to 3.80%, despite the single state variant achieving higher AULC under the fine schedule. Fixed equal allocation also increases the gap on five of the six pairs.

Unlike the original AULC only ablations, these experiments directly evaluate the contributions of the components of ALPS to schedule consistency. They show that within-batch integration contributes to schedule consistency, while learner-state conditioning supports a favorable trade-off between predictive performance and schedule sensitivity.

Validation supervision (W2, Q3)

We agree that ALPS uses validation labels beyond the checkpoint selection performed by all methods (App. C.2). To isolate this additional supervision, we evaluate ALPS-geo, which estimates the learner state using only geometric information, while retaining the use of the validation set for checkpoint selection as all other methods.

Across the five evaluated datasets, removing the predictive deficit (i.e., removing additional validation supervision) changes AULC by at most 0.47 points on four datasets, with a larger reduction of 1.73 on BRISC2025. ALPS-geo maintains a mean schedule gap of 1.84%, compared with 1.78% for ALPS. Thus, ALPS remains competitive without using validation labels to guide acquisition, although estimating the predictive deficit provides additional benefits, especially on BRISC2025. Full results are provided in App. D.5, Tab. 11.

Additionally, using only 10% of the validation labels for estimating the predictive deficit yields AULC close to full ALPS on BRISC2025 and CIFAR-100, although schedule gaps can differ (App. D.5, Tab. 12).

Adaptive baselines and novelty (W2, Q4)

We add TCM as an adaptive baseline under the same experimental protocol, using the transition rule recommended by the authors (Sec. 5.2, App. C.1, Tabs. 4 to 9).

Across the five evaluated datasets, ALPS achieves a smaller mean AULC deficit (0.82% vs. 1.24%) and a similar mean schedule gap (1.78% vs. 1.86%). TCM performs better on CIFAR-10 and SVHN, while ALPS has its strongest advantages on the medical datasets. For example, on BRISC2025, ALPS achieves 90.20 AULC and 0.83% schedule gap, compared with 87.83 and 2.55% for TCM.

Unlike the predefined transition used by TCM, ALPS adapts the allocation across three acquisition mechanisms using the estimated learner state and within-batch integration (Secs. 2 and 4). SelectAL and ALBL are also discussed in the revised related work (Sec. 2). However, a faithful SelectAL evaluation requires additional classifier training and cross-validation for strategy selection at each acquisition round, resulting in substantial computational overhead across our schedules. On the other hand, ALBL is originally formulated for sequential single-example acquisition, with bandit rewards updated after each query, and would require a nontrivial adaptation to our fixed batch protocol. Therefore, we prioritize the direct TCM comparison and do not claim empirical superiority over SelectAL or ALBL.

Performance trade-off and robustness (W2, S2)

We agree that the strongest gains of ALPS occur on medical images, while TypiClust leads on the natural image group. The revised analysis makes this distinction explicit (Sec. 5.2, Tab. 9).

Additional experiments show that ALPS outperforms

Thank you for the detailed response and the additional experiments. They address most of my questions.

However, the new results also make the advantage of ALPS less clear. Its schedule gaps are comparable to its seed variability on several datasets, and its mean gap is similar to that of TCM (1.78% vs. 1.86%). Without validation labels guiding acquisition, ALPS-geo has a larger AULC deficit than TCM (1.31% vs. 1.24%). In addition, TCM or BADGE performs best at extended budgets on three of four datasets and on ISIC2019 macro-F1. The benefit of state-conditioned allocation thus still seems limited to some settings, mainly the medical datasets.

Therefore, I keep my current rating for now.

Reviewer_sB34OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the overlooked impact of acquisition schedules in active learning, showing that different batch sizes and model update frequencies can lead to different predictive performance under the same annotation budget. The authors introduce schedule consistency and the schedule gap to quantify this sensitivity, and propose ALPS, a learner-state-aware scheduler that dynamically allocates acquisition batches across representativeness, coverage, and uncertainty based on geometric and predictive deficits. Experiments across five natural and medical image datasets reveal substantial schedule sensitivity among existing methods, while ALPS achieves the best overall balance between predictive performance and schedule consistency, particularly on medical imaging datasets. The results highlight the importance of considering learner state rather than annotation budget alone when designing active learning strategies.

Critical strengths
  1. The paper identified and systematized a problem of practical significance that is often not separately characterized in active learning evaluation: the same cumulative annotation budget does not imply the same learning trajectory or learner state. Based on this, the authors defined schedule consistency and schedule gap, elevating the acquisition schedule from an implementation detail to an independent evaluation dimension.

  2. The author systematically compared the schedule sensitivity under five natural and medical image datasets, ten active learning methods, and two acquisition schedules. Furthermore, the method design was validated through component ablation of ALPS, allocation trajectory analysis, ImageNet-200 extension experiments, and computational cost evaluation. Particularly importantly, the results indicated that a lower schedule gap does not necessarily correspond to higher prediction performance, while ALPS achieved a better balance between performance and stability overall.

Critical weaknesses
  1. The fairness of ALPS is affected by the additional verification supervision. The paper uses the classification error of the labeled validation set as the predictive deficit and directly adjusts the acquisition allocation based on this; however, these validation labels are not included in the annotation budget, while most baselines only use the same validation set for checkpoint selection. This weakens the strict fairness of "comparing acquisition efficiency under the same annotation budget", especially in the low-budget active learning scenarios. Although the authors demonstrated through an ablation study using only the geometric deficit that the method is not completely dependent on this signal, the current main results still cannot rule out the possibility that some advantages may come from the additional utilization of supervision information. Therefore, I believe this is the most significant issue that requires supplementary verification.

  2. The verification of the core "schedule robustness" claim is still rather limited. In the main experiment, only one pair of acquisition schedules was compared for each dataset, while the core contribution of the paper was to study the sensitivity of the schedule; moreover, the component ablation of ALPS only reported AULClog under a single schedule, and did not compare the schedule gap of each variant. Therefore, the existing evidence can demonstrate that different methods do indeed have schedule sensitivity, and also indicates that several components of ALPS contribute to prediction performance, but it cannot fully prove that the low schedule gap of ALPS is brought about by the stability of the state-conditioned design itself. For a paper with schedule consistency as its core theme, this is a relatively critical evidence gap.

Questions for authors
  1. ALPS uses the classification error of the labeled validation set as the predictive deficit and directly affects the acquisition allocation based on this. These validation labels are not included in the annotation budget. In contrast, the baseline mainly uses the same validation set for checkpoint selection. Could you further explain why this way of using supervision information still constitutes a fair comparison and report the complete main experimental results when not using the validation-label error (for example, only using the geometric deficit), rather than just the ablation study under a single schedule?

  2. The core proposition of the paper is schedule consistency. However, in the main experiments, each dataset only compared a pair of acquisition schedules, and the key component ablation in Table 1 only evaluated AULClog under a single schedule, without reporting the schedule gap of each variant. Could the author add intermediate batch-size schedules to at least one or two representative datasets, and compare the schedule gaps of key variants such as full ALPS, budget-conditioned/fixed allocation, and single-state?

3 comments

We thank the reviewer for the constructive suggestions. In response to the two main concerns, we conducted the suggested experiments and updated the paper accordingly. Specifically, we (1) evaluated ALPS without validation labels guiding acquisition (App. D.5, Tabs. 11 and 12) and (2) extended the evaluation to three acquisition schedules, including schedule gap comparisons for main ALPS variants (App. D.6, Tab. 13). These results strengthen our original conclusions, as detailed below.

Validation supervision (W1, Q1)

We agree that ALPS uses validation labels beyond the checkpoint selection performed by all methods (App. C.2). To isolate this additional supervision, we evaluate ALPS-geo, which estimates the learner state using only geometric information, while retaining the use of the validation set for checkpoint selection as all other methods.

Across the five evaluated datasets, removing the predictive deficit (i.e., removing additional validation supervision) changes AULC by at most 0.47 points on four datasets, with a larger reduction of 1.73 on BRISC2025. ALPS-geo maintains a mean schedule gap of 1.84%, compared with 1.78% for ALPS. Thus, ALPS remains competitive without using validation labels to guide acquisition, although estimating the predictive deficit provides additional benefits, especially on BRISC2025. Full results are provided in App. D.5, Tab. 11.

Additional schedules and component consistency (W2, Q2)

We introduce a third acquisition schedule on BRISC2025 (b=20,100,200b=20,100,200) and CIFAR-100 (b=100,500,1000b=100,500,1000), covering medical and natural-images classification. We evaluated all three schedule pairs for ALPS and its main variants (App. D.6, Tab. 13).

ALPS achieves smaller schedule gaps than BADGE and TypiClust (the baselines with the highest AULC on the selected datasets) across all six pairwise comparisons, with higher AULC on BRISC2025 and slightly lower on CIFAR-100. Importantly, removing within-batch integration increases the schedule gap on all six schedule pairs. On BRISC2025, the fine/coarse schedule gap increases from 0.83% to 3.80%, despite the single state variant achieving higher AULC under the fine schedule. Fixed equal allocation also increases the gap on five of the six pairs.

These additional experiments directly address the concern that the original ablations evaluated only predictive performance. They show that within-batch integration contributes to schedule consistency, while learner-state conditioning supports a favorable trade-off between predictive performance and schedule sensitivity.

I thank the authors for their careful response to my review comments and for adding relevant experiments addressing the two main concerns. I believe these additional experiments are highly targeted, especially the ALPS-geo experiment that does not use validation-set prediction errors, which effectively alleviates my previous concern that additional supervision information may affect the fairness of the comparison. In addition, the three newly added acquisition schedules and the ablation analysis of the schedule gap for key components further strengthen the empirical evidence for the paper’s claim of schedule consistency. In particular, removing within-batch integration increases the schedule gap in all six comparisons, providing direct evidence of the role of this component. Overall, the authors’ response substantively addresses the main concerns I raised, strengthening my confidence in the paper’s contributions and experimental conclusions. Although the current results are still insufficient to demonstrate the method’s general robustness to arbitrary sampling schedules, this limitation does not affect the paper’s main contribution under the settings studied.

We sincerely thank the reviewer for the follow-up and are glad that the additional experiments addressed the main concerns and strengthened the confidence in our contributions.

We agree that robustness to arbitrary acquisition schedules remains an open question, as discussed in our limitations (App. E).

Given the reviewer's positive reassessment, we would appreciate it if this could be reflected in an updated review, including a reconsideration of the overall rating.

Thank you again for the constructive feedback, which has helped us strengthen the message of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.