- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AHEAD, an active learning framework motivated by changes in sample difficulty during training. It predicts relative losses several epochs ahead and uses these predictions to train a representation-based importance function, which both reweights labeled training samples and ranks unlabeled candidates for annotation. Experiments on six datasets spanning images, text, and audio report improved training performance and competitive active-learning results, with ablations examining the effect of look-ahead supervision.
Critical strengths
-
AHEAD offers an interesting approach to loss-based active learning by accounting for how sample difficulty evolves during training. Its use of predicted future losses to supervise a shared importance function connects training-time reweighting with unlabeled sample selection, providing a coherent framework for incorporating anticipated difficulty into both stages.
-
The experimental evaluation is broad, covering six datasets across image, text, and audio classification with different backbone architectures. AHEAD improves over uniform training and achieves competitive active-learning performance in several settings. The accompanying horizon and warm-up ablations further explore when look-ahead supervision is beneficial, adding useful analysis beyond the main comparisons.
Critical weaknesses
-
AHEAD’s contribution is insufficiently distinguished from prior dynamics-aware and look-ahead acquisition methods. TiDAL and TOD already exploit training dynamics for acquisition, while LADA (NeurIPS 2021) anticipates augmentation effects and NTK-based look-ahead methods approximate candidate-induced model updates. These approaches differ from AHEAD’s prediction of relative loss at a specified horizon, but their omission leaves the methodological positioning incomplete. The paper needs to explain what this particular prediction target adds and provide targeted comparisons with the closest applicable alternatives. Contrasting current and future loss alone does not establish the added value over existing approaches that already consider training dynamics or subsequent operations.
-
The experimental evidence does not adequately substantiate the claimed acquisition advantage. Table 1 and the Round 0 differences in Figure 2 indicate training-side benefits, yet the evaluation does not isolate these from improvements in sample selection. Beyond this confounding, the central temporal-misalignment claim is not directly quantified, and the reported RankNet accuracy does not establish an advantage over current-loss rankings on unseen candidates. The acquisition-batch-size ablation also leaves unclear how the total annotation budget is controlled. Finally, the claim of highest final accuracy on CIFAR-10 and CIFAR-100 conflicts with Figure 2, where AutoAL with priors has higher final means. Together, these issues weaken confidence in both the explanation and the extent of AHEAD’s reported acquisition gains.
Questions for authors
-
Relation to prior work. How does AHEAD’s fixed-horizon loss target differ from the training-dynamics signals in TiDAL and TOD, and from the distinct look-ahead formulations in LADA and NTK-based acquisition? What evidence supports the added value of this particular target over the closest applicable alternatives?
-
Separating acquisition from training benefits. Were the initial labeled sets and model initializations shared across methods in Figure 2, and what explains the Round 0 performance differences? Does AHEAD retain an advantage when selected subsets are evaluated under a common training procedure, or when random acquisition is combined with AHEAD’s reweighting?
-
Validation of the future-difficulty signal. Can the authors quantify the claimed disagreement between current and future loss rankings? For Figure 3’s ranking accuracy, were predictions recorded before the corresponding future losses became available, and were the evaluated samples excluded from predictor training? A comparison with current-loss rankings, particularly for the highest-ranked candidates, would help establish whether forecasting provides additional acquisition-relevant information.
-
Interpretation of the batch-size ablation. Does – in Figure 3 denote the acquisition batch size or the training mini-batch size? If it denotes acquisition, what were the total annotation budget and number of rounds for each setting? Clarifying what was held fixed is necessary to distinguish an effect of acquisition granularity from differences in supervision or training.
-
Consistency of the final-performance claims. How should the claim in Section 4.2.2 that AHEAD achieves the highest final accuracy on CIFAR-10 and CIFAR-100 be reconciled with Figure 2, where AutoAL with priors appears to have higher final means? Could the authors provide the corresponding numerical results with uncertainty estimates and confirm that the endpoints use matching annotation budgets?
We sincerely thank you for your detailed review and constructive comments. We appreciate the opportunity to clarify the methodological distinctions and experimental settings.
Unlike prior methods that primarily use training dynamics to guide acquisition, AHEAD uses predicted future relative losses for both sample acquisition and training-time reweighting. Specifically, AHEAD jointly trains a look-ahead loss predictor, an importance function, and the backbone neural network, using predicted future losses to supervise sample reweighting during backbone training. The look-ahead mechanism in LADA considers future acquired samples, whereas AHEAD predicts sample difficulty at a fixed future training horizon. TOD constructs a target for the predictor to learn, whereas AHEAD uses observed future relative losses as delayed supervision. The distinction lies in both the prediction target and its use in acquisition and reweighting.
AutoAL and DISTIL use different codebases, which may account for the small differences observed at round 0. Within each codebase, the round-0 results are similar across methods, except on CIFAR-10, where the original training schedule was insufficient for convergence. We reran this experiment with additional training epochs and trials, and the updated results are provided in Figure 1 of the anonymous repository: https://anonymous.4open.science/r/AnonymousRepo-D90D/.
We emphasize that AHEAD combines sample acquisition with training-time reweighting through joint training of the look-ahead loss predictor, importance function, and backbone neural network. Consequently, the reported comparisons evaluate the overall procedure rather than isolating the effect of acquisition alone. We agree that the suggested controls would help separate these contributions. AutoAL also reported that its backbone-only baseline (referred to as the uniform backbone baseline) performs worse than several competing methods. To examine this setting, we ran a variant of AHEAD in which the evaluation model was trained as a backbone network using the acquired labels, without AHEAD's training-time reweighting. For example, the round 5 result for CIFAR-100 for this variant is 58.7018 ± 0.7493, which is substantially lower than the corresponding AHEAD result.
Ranking accuracy measures the predictor's ability to order samples by their future losses. It serves as a diagnostic of prediction quality rather than a direct measure of acquisition utility. We also report Pearson correlations between current and future losses for newly acquired samples before subsequent training. These results are provided in Table 1 of the linked repository.
B denotes the training mini-batch size, rather than the acquisition batch size. In this ablation, we train the model using the full dataset to assess the effect of mini-batch size on the reweighting procedure. We will make this distinction explicit in the figure caption.
We agree that the statement in Section 4.2.2 regarding the highest final accuracy on CIFAR-10 and CIFAR-100 is inaccurate. AHEAD does not achieve the highest final accuracy on both datasets. Our intended claim was that AHEAD achieves the best performance on most of the evaluated datasets, rather than all of them. We will revise the statement to be consistent with the reported results in Figure 2.