- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FractAL, a strategy-selection method for active learning with few acquisition rounds and large batches. Rather than choosing one strategy per round, it divides the labeling budget across a portfolio and updates the shares through online mirror descent using influence-based rewards from newly labeled sub-batches. This avoids additional retraining for strategy evaluation. Experiments across seven setups spanning classification, regression, and genetic perturbation prediction report gains over random sampling and favorable comparisons with existing strategy-selection methods, complemented by allocation diagnostics, component ablations, and runtime analysis.
Critical strengths
- FractAL addresses a well-motivated practical problem: choosing acquisition strategies when only a few experimental rounds are available. Combining fractional budget allocation with influence-based feedback allows several strategies to contribute within each batch, without additional retraining for strategy evaluation. This offers a simple and useful approach to adaptive strategy selection, particularly for applications where each acquisition round is costly.
- The evaluation spans classification, regression, and genetic perturbation prediction, with FractAL reporting gains over random sampling in all seven setups. Comparisons with equal allocation and adaptive selection methods, together with ablations on soft versus hard allocation and influence versus retraining rewards, provide useful evidence for the proposed design. The biological experiments also extend the empirical contribution beyond conventional active-learning benchmarks.
Critical weaknesses
-
The methodological contribution appears incremental relative to existing work. TAILOR (Zhang et al., NeurIPS 2023, Section 3.2) already selects multiple acquisition strategies within a batch and updates their feedback after joint annotation. This directly qualifies the paper’s characterization of prior strategy selection as committing an entire round to one strategy. Moreover, Proposition 1 applies the established group-influence formulation studied by Koh et al. (NeurIPS 2019), while the allocation rule is a standard KL-regularized exponentiated-gradient update. FractAL’s distinguishing feature is the use of retrospective validation-risk rewards to determine explicit budget shares. This is a potentially useful integration, particularly for regression, but the conceptual advance is narrower than the claimed introduction of a new soft-selection problem.
-
The experiments do not sufficiently establish the advantage of the proposed integration in the motivating few-round regime. The classification comparisons omit TAILOR, leaving out a directly relevant method that already combines strategy choices within a batch. The Equal Split and soft-versus-hard ablations are useful, but do not replace this comparison. Each setup also fixes the batch size and number of rounds, so there is no matched-budget evaluation showing how FractAL’s relative advantage changes as feedback opportunities decrease. Furthermore, the positive “worst-case gain” summarizes setup-level means rather than the frequency of underperformance across individual runs or splits. The results therefore support favorable average performance more clearly than an advantage over existing batch-aware selection methods or the claimed reliability when experimental rounds are scarce.
Questions for authors
- Novelty relative to prior work. How does FractAL’s soft-selection formulation differ from TAILOR’s batch-wise selection of multiple acquisition algorithms (Zhang et al., NeurIPS 2023)? Could the authors identify the specific methodological advance, distinguishing it from the established group-influence formulation of Koh et al. (NeurIPS 2019) and the standard mirror-descent update?
- Comparison with TAILOR. How does FractAL compare with TAILOR on the classification benchmarks under the same strategy portfolio, backbone, initial labeled set, and annotation budget? If TAILOR’s class-balancing objective makes such a comparison inappropriate, please explain the relevant differences in assumptions or objectives and how they affect the claimed advantage over existing batch-aware strategy selection.
- Evidence for the few-round advantage. Does FractAL’s relative advantage increase when the same total annotation budget is distributed over fewer, larger batches? A controlled comparison on the same dataset would help determine whether the method specifically addresses scarce feedback opportunities, rather than simply performing well at the particular batch sizes and round counts evaluated.
- Reliability across individual runs. Beyond the positive minimum gain across setup-level averages, how often does FractAL underperform Random on individual splits and seeds, and how large are those shortfalls? Could the authors report paired performance differences and uncertainty estimates that preserve variation across data splits? These results would help assess whether the favorable average performance also translates into the reduced selection risk emphasized in the paper.
3 comments
Author Response to the Review (2/2)
W2, Q4: Reliability across runs
Runs with the same split and seed share the initial labeled set across methods, so we computed paired differences against Random for every run, with uncertainty over both splits and seeds. Pooled across all seven setups (350 runs per method):
| Method | Runs below Random | Setups significantly below Random (of 7) |
|---|---|---|
| Equal Split | 37% | 1 |
| ALBL | 41% | 2 |
| UCB | 34% | 1 |
| SCRiBLe | 37% | 1 |
| SelectAL | 30% | 1 |
| AutoAL | 41% | 2 |
| FractAL | 19% | 0 |
Across all 350 runs, FractAL falls below Random least often, and it is the only method never significantly worse than Random on any setup. Its gain over Random is significant on five of seven setups. Individual-run shortfalls occur mainly on CIFAR-10, where single runs vary by several points for every method (FractAL's worst C1 run is 7.1 points below Random; the other methods' worst runs are 4.6–6.4 points below).
We hope our responses and the new results address your concerns. If so, we would be grateful if you would consider revisiting your score. We are happy to clarify anything further.
Approaching Rebuttal Deadline
Dear Reviewer F4Mj,
We would like to thank you again for your detailed review. As the discussion period ends on November 18, we wanted to gently check whether our responses have addressed your concerns. Following your suggestions, we added:
- a comparison with TAILOR, including allocation diagnostics and allocation plots
- experiments with fewer and larger batches under a fixed total budget
- a paired per-run reliability analysis that preserves variation across data splits
We would greatly appreciate any further feedback, and are happy to answer remaining questions before the deadline.
Author Response to the Review (1/2)
We thank the reviewer for the detailed review and for recognizing the well-motivated practical problem that our method addresses, the simplicity and efficiency of our method and the breadth of the evaluation across conventional classification, regression tasks as well as the uncommon biological experiments. Below, we aim to address each of your concerns.
W1, Q1, Q2: Novelty and TAILOR method
We agree that the individual components of FractAL exist already. Proposition 1 is the group-influence result of Koh et al. (2019) applied to strategy-level groups, and the update is standard online mirror descent (exponentiated gradient). Our contribution is how they fit together in the context of active learning. Splitting each batch across strategies, and using influence functions as a cheap proxy for the change in validation risk attributable to each strategy's sub-batch, gives every strategy a performance-based reward every round. This turns strategy selection from a bandit problem, where one scalar score per batch cannot be split across strategies, into an approximately full-information one, for which exponentiated gradient is the natural update. We will acknowledge and clarify this more explicitly in the camera-ready version along with the citations mentioned above.
We sincerely thank the reviewer for mentioning the TAILOR baseline. As you rightly pointed out, TAILOR also attributes reward per strategy, but its reward depends only on the labels of acquired points: it measures class balance rather than model improvement and requires class labels. FractAL's reward works for any differentiable loss, which is why it applies to the five regression and perturbation screen setups in our paper where TAILOR does not.
We ran TAILOR on the CIFAR-10 C1 and C2 setups with the same portfolio, backbone, initial set, budget, and batch formation as all other methods. Terminal accuracy is reported below (mean ± s.e. over 5 splits and 10 runs per split):
TAILOR is the strongest baseline on CIFAR-10 C1 and ties FractAL on C2. FractAL and TAILOR are the only methods above Random on both configurations, and both attribute reward to each strategy within the batch.
Our allocation diagnostics (new Appendix E) show where the two differ. TAILOR prunes the weakest strategy almost as reliably as FractAL, but it does not concentrate budget on the strongest one:
TAILOR's Best-Set Overlap is below chance on C1 and close to chance on C2. This is consistent with its reward tracking class balance rather than model improvement. It can identify a strategy that collects unbalanced data, but not which of the remaining strategies most improves the model. Per-strategy credit assignment is what lets both methods exceed Random; a performance-based reward is what lets FractAL also identify the strongest strategy. We have added few sentences about TAILOR in the background, performance numbers in table 2, allocation diagnostics in Appendix E and allocation plots in Appendix K figures 5-6 (panel d).
W2, Q3: Fewer, larger batches
We thank the reviewer for this constructive suggestion. With the total budget fixed at 5,000 labels (|D_init| = B), we varied the number of rounds on a classification (CIFAR-10 C1) and a regression setup (KEGG):
FractAL is the best adaptive method at every round count, and on C1 the only one above Random, while the strongest baseline changes with the regime. As rounds decrease, FractAL's margin stays roughly stable and narrows slightly at T = 5, where fewer updates and a larger fixed initial set leave every method less room to adapt. On KEGG the margins are small. We provide the full tables in Appendix I.6. In the interest of time, we did not run AutoAL for these experiments but we will include it in the camera-ready version.