acceptodds
Under review as a conference paper at ICLR 2027

Soft Strategy Selection for Batch-Mode Active Learning

Abstract

Real-world deployment of active learning (AL) typically forces practitioners to choose an acquisition strategy before any data is labeled. This is a daunting task: strategy performance varies widely across settings (e.g., datasets and surrogate model architectures) and cannot be assessed without deployment. Existing strategy selection methods explore one strategy from a portfolio at each round and identify the optimal one using bandit feedback or model retraining. Many acquisition rounds are therefore spent exploring strategies rather than collecting the most informative data. Such overhead is a major barrier to AL-driven design of high-throughput experiments, such as genetic perturbation screens and directed evolution, where AL runs consist of only a few rounds with large batch sizes. This regime permits a natural alternative: acquiring data using multiple AL strategies within a single batch. We refer to this as soft strategy selection and introduce FractAL, a method specifically designed for this task. FractAL infers a per-strategy reward using influence-function-based data attribution, which requires no additional retraining, and then computes budget shares for each strategy in the portfolio using online mirror descent. We benchmark FractAL across seven active learning setups spanning classification, regression, and genetic perturbation effect prediction. The results highlight that strategy selection is a hard problem: every existing method performs worse than random sampling on at least one setup. FractAL, however, matches or outperforms every baseline, including random sampling, on all seven setups. Its allocations concentrate budget on the strongest strategies in the portfolio while pruning the weakest. FractAL is therefore a reliable choice for real-world deployments, where the optimal strategy is unknown in advance, an important step towards making active learning practical for high-throughput experiments and modern scientific discovery.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Strategy Selection, Active Learning, High-Throughput Genetic Screens, Influence Functions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_F4MjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FractAL, a strategy-selection method for active learning with few acquisition rounds and large batches. Rather than choosing one strategy per round, it divides the labeling budget across a portfolio and updates the shares through online mirror descent using influence-based rewards from newly labeled sub-batches. This avoids additional retraining for strategy evaluation. Experiments across seven setups spanning classification, regression, and genetic perturbation prediction report gains over random sampling and favorable comparisons with existing strategy-selection methods, complemented by allocation diagnostics, component ablations, and runtime analysis.

Critical strengths
  1. FractAL addresses a well-motivated practical problem: choosing acquisition strategies when only a few experimental rounds are available. Combining fractional budget allocation with influence-based feedback allows several strategies to contribute within each batch, without additional retraining for strategy evaluation. This offers a simple and useful approach to adaptive strategy selection, particularly for applications where each acquisition round is costly.
  2. The evaluation spans classification, regression, and genetic perturbation prediction, with FractAL reporting gains over random sampling in all seven setups. Comparisons with equal allocation and adaptive selection methods, together with ablations on soft versus hard allocation and influence versus retraining rewards, provide useful evidence for the proposed design. The biological experiments also extend the empirical contribution beyond conventional active-learning benchmarks.
Critical weaknesses
  1. The methodological contribution appears incremental relative to existing work. TAILOR (Zhang et al., NeurIPS 2023, Section 3.2) already selects multiple acquisition strategies within a batch and updates their feedback after joint annotation. This directly qualifies the paper’s characterization of prior strategy selection as committing an entire round to one strategy. Moreover, Proposition 1 applies the established group-influence formulation studied by Koh et al. (NeurIPS 2019), while the allocation rule is a standard KL-regularized exponentiated-gradient update. FractAL’s distinguishing feature is the use of retrospective validation-risk rewards to determine explicit budget shares. This is a potentially useful integration, particularly for regression, but the conceptual advance is narrower than the claimed introduction of a new soft-selection problem.

  2. The experiments do not sufficiently establish the advantage of the proposed integration in the motivating few-round regime. The classification comparisons omit TAILOR, leaving out a directly relevant method that already combines strategy choices within a batch. The Equal Split and soft-versus-hard ablations are useful, but do not replace this comparison. Each setup also fixes the batch size and number of rounds, so there is no matched-budget evaluation showing how FractAL’s relative advantage changes as feedback opportunities decrease. Furthermore, the positive “worst-case gain” summarizes setup-level means rather than the frequency of underperformance across individual runs or splits. The results therefore support favorable average performance more clearly than an advantage over existing batch-aware selection methods or the claimed reliability when experimental rounds are scarce.

Questions for authors
  1. Novelty relative to prior work. How does FractAL’s soft-selection formulation differ from TAILOR’s batch-wise selection of multiple acquisition algorithms (Zhang et al., NeurIPS 2023)? Could the authors identify the specific methodological advance, distinguishing it from the established group-influence formulation of Koh et al. (NeurIPS 2019) and the standard mirror-descent update?
  2. Comparison with TAILOR. How does FractAL compare with TAILOR on the classification benchmarks under the same strategy portfolio, backbone, initial labeled set, and annotation budget? If TAILOR’s class-balancing objective makes such a comparison inappropriate, please explain the relevant differences in assumptions or objectives and how they affect the claimed advantage over existing batch-aware strategy selection.
  3. Evidence for the few-round advantage. Does FractAL’s relative advantage increase when the same total annotation budget is distributed over fewer, larger batches? A controlled comparison on the same dataset would help determine whether the method specifically addresses scarce feedback opportunities, rather than simply performing well at the particular batch sizes and round counts evaluated.
  4. Reliability across individual runs. Beyond the positive minimum gain across setup-level averages, how often does FractAL underperform Random on individual splits and seeds, and how large are those shortfalls? Could the authors report paired performance differences and uncertainty estimates that preserve variation across data splits? These results would help assess whether the favorable average performance also translates into the reduced selection risk emphasized in the paper.
3 comments
AuthorsOpenReview ↗

Author Response to the Review (1/2)

We thank the reviewer for the detailed review and for recognizing the well-motivated practical problem that our method addresses, the simplicity and efficiency of our method and the breadth of the evaluation across conventional classification, regression tasks as well as the uncommon biological experiments. Below, we aim to address each of your concerns.


W1, Q1, Q2: Novelty and TAILOR method

We agree that the individual components of FractAL exist already. Proposition 1 is the group-influence result of Koh et al. (2019) applied to strategy-level groups, and the update is standard online mirror descent (exponentiated gradient). Our contribution is how they fit together in the context of active learning. Splitting each batch across strategies, and using influence functions as a cheap proxy for the change in validation risk attributable to each strategy's sub-batch, gives every strategy a performance-based reward every round. This turns strategy selection from a bandit problem, where one scalar score per batch cannot be split across strategies, into an approximately full-information one, for which exponentiated gradient is the natural update. We will acknowledge and clarify this more explicitly in the camera-ready version along with the citations mentioned above.

We sincerely thank the reviewer for mentioning the TAILOR baseline. As you rightly pointed out, TAILOR also attributes reward per strategy, but its reward depends only on the labels of acquired points: it measures class balance rather than model improvement and requires class labels. FractAL's reward works for any differentiable loss, which is why it applies to the five regression and perturbation screen setups in our paper where TAILOR does not.

We ran TAILOR on the CIFAR-10 C1 and C2 setups with the same portfolio, backbone, initial set, budget, and batch formation as all other methods. Terminal accuracy is reported below (mean ± s.e. over 5 splits and 10 runs per split):

CIFAR-10 (C1) Acc.CIFAR-10 (C2) Acc.
Random69.4 ± 0.169.4 ± 0.1
TAILOR69.7 ± 0.269.8 ± 0.1
FractAL70.2 ± 0.269.8 ± 0.2

TAILOR is the strongest baseline on CIFAR-10 C1 and ties FractAL on C2. FractAL and TAILOR are the only methods above Random on both configurations, and both attribute reward to each strategy within the batch.

Our allocation diagnostics (new Appendix E) show where the two differ. TAILOR prunes the weakest strategy almost as reliably as FractAL, but it does not concentrate budget on the strongest one:

C1 Best-SetC1 Worst-SetC2 Best-SetC2 Worst-Set
Chance0.330.330.250.13
TAILOR0.21 ± 0.030.82 ± 0.020.29 ± 0.020.60 ± 0.04
FractAL0.79 ± 0.020.89 ± 0.020.46 ± 0.010.61 ± 0.01

TAILOR's Best-Set Overlap is below chance on C1 and close to chance on C2. This is consistent with its reward tracking class balance rather than model improvement. It can identify a strategy that collects unbalanced data, but not which of the remaining strategies most improves the model. Per-strategy credit assignment is what lets both methods exceed Random; a performance-based reward is what lets FractAL also identify the strongest strategy. We have added few sentences about TAILOR in the background, performance numbers in table 2, allocation diagnostics in Appendix E and allocation plots in Appendix K figures 5-6 (panel d).


W2, Q3: Fewer, larger batches

We thank the reviewer for this constructive suggestion. With the total budget fixed at 5,000 labels (|D_init| = B), we varied the number of rounds on a classification (CIFAR-10 C1) and a regression setup (KEGG):

SetupT × BRandomBest baselineFractAL
C1 (accuracy ↑)10 × 50069.468.3 (SelectAL)70.2
8 × 62569.668.1 (UCB)70.2
5 × 100069.268.8 (UCB)69.7
KEGG (MSE ×10⁻¹ ↓)10 × 5000.790.59 (SelectAL)0.58
8 × 6250.810.62 (UCB, SelectAL)0.60
5 × 10000.760.62 (UCB)0.61

FractAL is the best adaptive method at every round count, and on C1 the only one above Random, while the strongest baseline changes with the regime. As rounds decrease, FractAL's margin stays roughly stable and narrows slightly at T = 5, where fewer updates and a larger fixed initial set leave every method less room to adapt. On KEGG the margins are small. We provide the full tables in Appendix I.6. In the interest of time, we did not run AutoAL for these experiments but we will include it in the camera-ready version.

AuthorsOpenReview ↗

Author Response to the Review (2/2)

W2, Q4: Reliability across runs

Runs with the same split and seed share the initial labeled set across methods, so we computed paired differences against Random for every run, with uncertainty over both splits and seeds. Pooled across all seven setups (350 runs per method):

MethodRuns below RandomSetups significantly below Random (of 7)
Equal Split37%1
ALBL41%2
UCB34%1
SCRiBLe37%1
SelectAL30%1
AutoAL41%2
FractAL19%0

Across all 350 runs, FractAL falls below Random least often, and it is the only method never significantly worse than Random on any setup. Its gain over Random is significant on five of seven setups. Individual-run shortfalls occur mainly on CIFAR-10, where single runs vary by several points for every method (FractAL's worst C1 run is 7.1 points below Random; the other methods' worst runs are 4.6–6.4 points below).


We hope our responses and the new results address your concerns. If so, we would be grateful if you would consider revisiting your score. We are happy to clarify anything further.

AuthorsOpenReview ↗

Approaching Rebuttal Deadline

Dear Reviewer F4Mj,

We would like to thank you again for your detailed review. As the discussion period ends on November 18, we wanted to gently check whether our responses have addressed your concerns. Following your suggestions, we added:

  • a comparison with TAILOR, including allocation diagnostics and allocation plots
  • experiments with fewer and larger batches under a fixed total budget
  • a paired per-run reliability analysis that preserves variation across data splits

We would greatly appreciate any further feedback, and are happy to answer remaining questions before the deadline.

Reviewer_tMN7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FractAL, a batch-mode active learning framework that allocates labeling budgets across multiple acquisition strategies. It uses influence functions to estimate strategy contributions and mirror descent to update allocations. Experiments across seven setups demonstrate improvements over adaptive baselines.

Critical strengths
  1. Addresses a practical and important problem in active learning.
  2. Combines efficient influence-based attribution with adaptive budget allocation.
Critical weaknesses
  1. The novelty claim overlooks earlier probabilistic strategy-blending approaches. The novelty framing is somewhat overstated. ALBL (Hsu & Lin, 2015) already combines strategy-specific query distributions through adaptive probabilistic weights. Moreover, the paper's ALBL baseline assigns each entire batch to one strategy, potentially disadvantaging it. A mixed-batch ALBL comparison would better isolate FractAL's contribution.

  2. The ALBL baseline uses whole-batch strategy selection, potentially creating an unfair comparison.

  3. Influence-function approximations lack strong validation across different tasks. Could the authors report influence-versus-retraining reward correlations for regression and biological tasks, as done for CIFAR-10 C1 in Table 15, to establish the reliability of strategy attribution across different model architectures and tasks?

  4. Limited independent datasets and statistical analysis weaken generalization claims.

Questions for authors
  1. Why does the ALBL implementation select one strategy per batch rather than sampling from a mixture of strategies? Could you evaluate a mixed-batch variant?

  2. How are influence rewards calculated for unselected strategies in the hard-allocation ablation?

  3. How well do approximate influence rewards correlate with actual marginal improvements on regression and biological tasks?

  4. Could selecting only perturbations with significant measured effects bias the candidate pool and overestimate performance?

  5. Could repeated use of the same small validation set for checkpoint selection and strategy attribution cause overfitting?

4 comments
AuthorsOpenReview ↗

Author Response to the Review (1/2)

We thank the reviewer for the careful review and for recognizing that FractAL addresses a practical and important problem and uses efficient rewards for adaptive budget allocation. In the following, we aim to address all your concerns:


W1, W2, Q1: ALBL and mixed batches

Thank you for raising this. We agree that ALBL does mix strategies. In the original method, queries are made one at a time. Specifically, Exp4.P combines the strategies' recommendations using its current weights, the next point to query is drawn from this mixture, and the resulting reward is credited to the strategies that recommended it. Over many single queries, the labeled set therefore mixes strategies. This per-query credit relies on observing a reward after every query, which the batch setting does not provide, since the model is retrained and evaluated once per batch. Our batch adaptation therefore assigns each batch to one sampled strategy, so that the observed reward can be credited to it.

A mixed-batch variant would allocate the batch according to ALBL's weights, but it would still observe one scalar validation reward per batch, which cannot be attributed to the strategies inside it. SCRiBLe is the principled version of this baseline (a soft allocation learned from scalar feedback), and it trails FractAL on all seven setups. We also added TAILOR (Zhang et al., 2023), which mixes strategies within a batch and gives each one a label-based reward (accuracy):

CIFAR-10 (C1) Acc.CIFAR-10 (C2) Acc.
Random69.4 ± 0.169.4 ± 0.1
SCRiBLe67.7 ± 0.269.2 ± 0.1
TAILOR69.7 ± 0.269.8 ± 0.1
FractAL70.2 ± 0.269.8 ± 0.2

FractAL and TAILOR are the only methods above Random on both CIFAR-10 configurations, and both attribute reward to each strategy within the batch. This supports our view that per-strategy credit assignment, rather than mixing itself, is what matters. TAILOR's reward requires class labels, so it does not apply to our regression setups.


Q2: Hard allocation

A strategy that receives no budget in a round is assigned a reward of 0 for that round, so its smoothed reward decays toward zero (it is multiplied by 1 − α); only the selected strategy receives a new influence-based reward. The hard variant therefore observes only one strategy per round, whereas FractAL observes all of them every round. We have added this detail in Appendix I.4.


W3, Q3: Influence vs. retraining

Similar to the format for CIFAR-10 C1 dataset, we computed the per-round agreement between influence-based and retraining-based rewards on the remaining setups: Spearman rank correlation, and the overlap between the best (and worst) strategy sets ranked by each reward. Chance levels are in parentheses; full per-round results are in the Appendix I.3. We present overall numbers for the setups below:

KEGGSARCOSDiamondsB1B2
Spearman ρ (0)0.290.350.060.620.56
Best-set overlap0.61 (0.50)0.64 (0.50)0.52 (0.50)0.58 (0.25)0.68 (0.25)
Worst-set overlap0.32 (0.25)0.81 (0.75)0.52 (0.50)0.79 (0.50)0.68 (0.50)

Agreement is high on the biological tasks, where the surrogate has a single layer and the Hessian is exact, and lower on the regression tasks. On Diamonds, the pure strategies perform almost identically, so neither reward can separate them. Despite these differences, the two rewards lead to comparable final performance on every setup (Table 15), and on B2 influence performs better (0.3642 vs. 0.3514).


Q4: Filtered perturbations

The filter is applied once to the pool, before any method sees data, so all methods face the same candidates and the comparison between them is unaffected. Perturbations without a measurable effect carry little signal for the differential gene expression task we evaluate.


Q5: Validation reuse

All methods use the same validation set for checkpoint selection, and the bandit baselines also use it for their rewards, so reuse gives FractAL no relative advantage. All reported results are on a separate held-out test set, so any overfitting to the validation set would lower FractAL's reported performance rather than inflate it. Furthermore, FractAL's allocation also has few degrees of freedom (a point on the K-simplex, updated at most nine times), which limits how much it can fit the validation set.

AuthorsOpenReview ↗

Author Response to the Review (2/2)

W4: Statistics

Our evaluation covers six datasets across three task types (image classification, tabular regression, and multi-task perturbation-effect prediction) and three surrogate architectures. For strategy selection, what matters is that the best strategy changes across settings, and it does in our evaluations. CoreSet is the worst strategy on CIFAR-10 but among the strongest on the regression and perturbation tasks, and ProbCover is the best on KEGG and SARCOS but the worst on Diamonds. Yet, FractAL outperforms other adaptive strategy selection frameworks across all these setups. We, however, agree that more datasets would further strengthen the conclusions and support generalizability.

To strengthen the statistical analysis, we computed paired differences against Random for every run (runs with the same split and seed share the initial labeled set across methods), with uncertainty over both splits and seeds. Pooled across all seven setups (350 runs per method):

MethodRuns below RandomSetups significantly below Random (of 7)
Equal Split37%1
ALBL41%2
UCB34%1
SCRiBLe37%1
SelectAL30%1
AutoAL41%2
FractAL19%0

Across all 350 runs, FractAL falls below Random least often, and it is the only method never significantly worse than Random on any setup.


We hope our responses and the new results address your concerns. If so, we would be grateful if you would consider revisiting your score. We are happy to clarify anything further.

AuthorsOpenReview ↗

Approaching Rebuttal Deadline

Dear Reviewer tMN7,

We want to thank you again for your careful review. As the discussion period ends on November 18, we wanted to gently check whether our responses have addressed your concerns. In addition to clarification responses, based on your questions, we:

  • clarified the ALBL comparison and added TAILOR as a mixed-batch baseline
  • reported influence-versus-retraining agreement on all regression and biological setups
  • added a paired per-run reliability analysis across all setups.

We would greatly appreciate any further feedback, and are happy to answer remaining questions before the deadline.

Reviewer_tMN7OpenReview ↗

Reviewer Response

Thank you for the additional experiments and clarifications. They address my main concerns sufficiently, and I’m raising my score to weak acceptance. Please still clarify the “exact Hessian” calculation and whether biological filtering requires knowing target outcomes in advance. The attribution and superiority claims should also be qualified given the weak regression correlations and the new baseline results.

Reviewer_eWoNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies adaptive acquisition strategy selection for batch-mode active learning, particularly for few acquisition rounds and relatively large batch sizes. Instead of committing the entire acquisition budget to one strategy per round, the authors propose FractAL, which distributes each batch across multiple acquisition strategies and adaptively updates their budget allocations. FractAL consists of two main components: (1) influence-function-based attribution to estimate the contribution of each strategy's acquired samples to validation performance without additional model retraining, and (2) an online mirror descent procedure to update strategy allocations based on smoothed reward estimates.

The method is evaluated across seven active learning configurations covering image classification, regression, and genetic perturbation effect prediction. The experimental results show that FractAL consistently matches or outperforms competing adaptive strategy selection approaches, while maintaining relatively low computational overhead. Additional ablation studies investigate the effects of reward estimation, allocation optimization, and fractional allocation.

Critical strengths
  1. A well-motivated problem with a practical method. The paper addresses an important limitation of existing active learning strategy selection approaches: committing an entire acquisition batch to a single strategy can be inefficient when only a few rounds of feedback are available. FractAL offers a meaningful alternative by allocating acquisition budgets across multiple strategies within each round. Its influence-based reward estimation also avoids the additional model retraining required by some competing approaches.
  2. The experimental evaluation shows the effectiveness of the proposed method. The experimental evaluation covers seven configurations involving classification, regression, and biological perturbation prediction. FractAL consistently achieves the best performance among adaptive strategy selection methods and avoids underperforming random sampling across the evaluated configurations. The allocation diagnostics further demonstrate that the method generally assigns larger budgets to stronger strategies while reducing the influence of weaker ones. The ablation studies provide additional evidence supporting the contributions of fractional allocation and influence-based reward estimation. But I have some other concerns in the evaluation, as stated in the Weaknesses below.
Critical weaknesses
  1. Limited novelty beyond the integration of existing techniques.

The two central components of FractAL, an influence-function-based reward estimator and online mirror descent for probability-simplex optimization, are both well-established techniques. Although their combination provides a useful solution for adaptive active learning strategy allocation, the paper offers limited new theoretical understanding of why this particular combination should reliably outperform alternative allocation mechanisms. Proposition 1 largely follows the standard influence-function formulation, extended to strategy-specific groups of samples. The contribution is therefore primarily algorithmic integration and empirical validation rather than a fundamentally new optimization or attribution methodology.

  1. Limited validation of robustness and generalizability under challenging conditions.

The experiments demonstrate consistent improvements, but they largely rely on fixed acquisition budgets, relatively small strategy portfolios, and a limited selection of surrogate architectures. In particular, the biological experiments use a single-layer MLP and batches of only 35 samples. They do not fully establish effectiveness in the large-scale, computationally demanding settings which motivate the paper.

Questions for authors
  1. How sensitive is FractAL to the last-layer Hessian approximation and the damping parameter? Have you evaluated whether reward rankings or final allocation performance change substantially under different damping strengths or surrogate model architectures?
  2. How does FractAL perform when the number of acquisition strategies increases or when the acquisition batch size and number of rounds change substantially? Since the method is motivated by few-round, large-batch active learning, understanding its behavior across different budget regimes would help clarify the range of settings where fractional allocation provides a meaningful advantage.
  3. Could you further quantify when adaptive allocation provides substantial benefits over equal splitting or other simple portfolio allocation policies? For example, how does FractAL compare against a simple explore-then-commit or successive-elimination baseline: equal split in round 1, drop strategies whose validation-based or influence-based score is clearly worst, then split equally among the survivors?
4 comments
AuthorsOpenReview ↗

Author Response to the Review (1/2)

We thank the reviewer for the thoughtful review and for recognizing that our problem formulation is well-motivated, FractAL is a practical method to address it without any additional retraining, and is consistently the best across the seven setups. We address each of your concerns below.


W1: Novelty

We agree that the individual components of FractAL exist already. Proposition 1 is the group-influence result of Koh et al. (2019) applied to strategy-level groups, and the update is the standard online mirror descent. Our contribution is how they fit together in the context of Active Learning. Splitting each batch across strategies and attributing the influence as a cheap proxy of change in validation risk to each strategy's sub-batch gives every strategy a performance-based reward every round. This turns strategy selection from a bandit problem, where one scalar score per batch cannot be split across strategies, into an approximately full-information one, for which exponentiated gradient is the natural update. We will acknowledge and clarify this more explicitly in the camera-ready version along with the citations mentioned above. Our ablations show that each of these two pieces matter. SCRiBLe uses the soft allocation with only a scalar feedback and trails FractAL on all seven setups. On the other hand, the hard-allocation variant (App I.4, Table 18) uses the same influence rewards but samples from and observes a new reward for only one strategy per round, and underperforms on five of seven setups.


Q1: Damping and last-layer Hessian approximation

On B1/B2 the surrogate has a single layer, so the Hessian is exact, allowing the isolation of the effect of damping parameter. We report AUPRC and Best-/Worst-Set Overlap as described in the paper:

λB1 AUPRCB1 Best / WorstB2 AUPRCB2 Best / Worst
0.0010.43660.76 / 0.650.35770.55 / 0.59
0.01 (Used in paper)0.43630.70 / 0.620.36420.57 / 0.61
0.50.43640.71 / 0.630.36400.51 / 0.61

Results are unchanged from λ = 0.01 to 0.5, and allocations are stable across all three values: both overlaps change by at most 0.06 and stay well above chance. At λ = 0.001, B2's AUPRC becomes noisier even though its allocations barely change. We used λ = 0.01 in every setup without tuning. We have added this ablation to the revised manuscript (Appendix I.7).

Across the three types of tasks, we use three different surrogate models, ResNet 18 for image classification, 3 layer MLP for the regression and 1 layer MLP for the biological perturbation datasets. The consistency of FractAL's allocation diagnostics with the pure strategies' empirical performance and superior terminal performance highlights the quality and robustness of our performance and reward rankings to different surrogate architecturs. Specifically, for the last-layer approximation, influence rewards agree with the retraining-based rewards they approximate (Spearman ρ = 0.70–0.80 on CIFAR-10, Table 16), and swapping one for the other gives comparable performance on all seven setups (Table 15).


Q2: Budget regimes and portfolio size

We thank the reviewer for this constructive suggestion. With the total budget fixed at 5,000 labels (|D_init| = B), we varied the number of rounds on a classification (CIFAR-10 C1) and a regression setup (KEGG):

SetupT × BRandomBest baselineFractAL
C1 (accuracy ↑)10 × 50069.468.3 (SelectAL)70.2
8 × 62569.668.1 (UCB)70.2
5 × 100069.268.8 (UCB)69.7
KEGG (MSE ×10⁻¹ ↓)10 × 5000.790.59 (SelectAL)0.58
8 × 6250.810.62 (UCB, SelectAL)0.60
5 × 10000.760.62 (UCB)0.61

FractAL is the best adaptive method at every round count, and on C1 the only one above Random, while the strongest baseline changes with the regime. We provide the full tables in Appendix I.6. In the interest of time, we did not run AutoAL for these experiments but we will include it in the camera-ready version.

For portfolio size, C2 uses K = 8 in contrast to other setups with K= 3 or 4, and FractAL's cost barely grows with K. Each new strategy in the pool is just one inverse-Hessian-vector product per round. The biological batch size (B = 35) reflects the size of current paired datasets (514 and 709 perturbations); six rounds still cover 40–50% of the pool, which is the few-round regime we target.

AuthorsOpenReview ↗

Author Response to the Review (2/2)

Q3: Successive Elimination Baseline

We thank the reviewer for suggesting this baseline. Here are the results for it across all our setups. Each round it splits the budget equally among the remaining strategies and eliminates the one with the lowest influence-based reward, until a single strategy remains and receives the full budget for the remaining AL run.

SetupRandomSuccessive eliminationFractAL
CIFAR-10 C1 (accuracy ↑)69.4 ± 0.170.8 ± 0.270.2 ± 0.2
CIFAR-10 C2 (accuracy ↑)69.4 ± 0.170.0 ± 0.169.8 ± 0.2
KEGG (MSE ×10⁻¹ ↓)0.79 ± 0.020.61 ± 0.020.58 ± 0.01
SARCOS (MSE ×10⁻¹ ↓)0.79 ± 0.010.78 ± 0.010.74 ± 0.01
Diamonds (MSE ×10⁻¹ ↓)0.11 ± 0.010.10 ± 0.000.08 ± 0.01
B1 (AUPRC ↑)0.4165 ± 0.00110.4338 ± 0.00090.4363 ± 0.0006
B2 (AUPRC ↑)0.3403 ± 0.00450.3605 ± 0.00470.3642 ± 0.0013

FractAL is better on five of the seven setups, clearly so on SARCOS, Diamonds, and B1. Successive elimination is better on C1, where strategy quality differs sharply, as it eliminates CoreSet and then Random within two rounds and commits the remaining budget to ProbCover, nearly matching ProbCover's individual empirical performance (71.2). On C2, the two methods are within noise (70.0 vs. 69.8). Because it uses FractAL's influence-based reward, this also shows that the reward identifies the right ranking quickly when strategies differ clearly. When strategies are closer in performance, as on the regression and perturbation tasks, each elimination relies on a single round of noisy rewards and cannot be undone, and successive elimination falls behind. For example, on SARCOS and Diamonds it barely improves on Random's performance. FractAL's gradual updates, in contrast, give up a little in the easily distinguishable case in exchange for robustness when the ranking is uncertain. We include the details of the baseline and the full results in Appendix I.5 of the revised manuscript.


We hope our responses and the new results address your concerns. If so, we would be grateful if you would consider revisiting your score. We are happy to clarify anything further.

AuthorsOpenReview ↗

Approaching Rebuttal Deadline

Dear Reviewer eWoN,

We sincerely thank you for your thoughtful review. As the discussion period ends on November 18, we wanted to gently check whether our responses have addressed your concerns. Following your suggestions, we added:

  • a damping sensitivity analysis on the two biological tasks where the Hessian is exact
  • experiments with fewer rounds and larger batches under a fixed total budget
  • a comparison with successive elimination baseline across all seven setups.

We would greatly appreciate any further feedback, and are happy to answer remaining questions before the deadline.

Thank you for the rebuttal. I appreciate the new analyses on damping sensitivity, different budget regimes, and the successive-elimination baseline. These results can help clarify the robustness and practical effectiveness of the proposed method. However, my main concern regarding methodological novelty remains. The core components are based on established techniques, and the contribution is primarily their integration into the active learning setting. I am still not sufficiently convinced of its technical contribution. Therefore, I will maintain my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summarizing the Author-Reviewer Discussion

Dear Area Chair,

As the author-reviewer discussion period comes to an end, we thank the reviewers for recognizing our problem as well-motivated and practical (eWoN, tMN7, F4Mj) and important (tMN7), and our method as a simple and useful approach (F4Mj) that combines efficient influence-based attribution with adaptive budget allocation (tMN7) without additional retraining (eWoN, F4Mj). The reviewers also noted that FractAL consistently achieves the best performance among adaptive methods without underperforming random sampling (eWoN), that the biological experiments extend the evaluation beyond conventional active learning benchmarks (F4Mj), and that the allocation diagnostics and ablations provide useful evidence for the design (eWoN, F4Mj).

To address the reviewers' concerns, we added two new baselines (TAILOR and successive elimination), experiments with fewer and larger batches under a fixed budget, a damping sensitivity analysis, influence-versus-retraining agreement on all setups, and a paired per-run reliability analysis. The revisions are in blue in the revised PDF. Reviewer tMN7 raised their score; Reviewer eWoN found that the new analyses clarify FractAL's robustness and practical effectiveness, but kept their novelty concern, which we address in our response and below. Reviewer F4Mj has not yet responded.

New results

  • TAILOR (F4Mj, tMN7; Table 2, Appendix E, Figs 5–6 (panel d) in Appendix K): TAILOR is the strongest baseline on CIFAR-10. FractAL is higher on C1 (70.2 vs. 69.7) and ties on C2 (69.8). TAILOR prunes the weakest strategy but does not identify the strongest (Best-Set Overlap 0.21 vs. 0.79 for FractAL on C1), consistent with its reward tracking class balance rather than model improvement.
  • Fewer, larger batches under a fixed budget (F4Mj, eWoN; Appendix I.6): On CIFAR-10 C1 and KEGG with 10, 8, or 5 rounds, FractAL is the best adaptive method at every round count, and on C1 the only one above Random. Its margin stays roughly stable even with half the rounds.
  • Successive elimination (eWoN; Appendix I.5): This baseline eliminates the weakest strategy (by influence reward) each round until one remains, which then receives the full budget thereafter. FractAL is better on five of seven setups; successive elimination is better on C1 and within noise on C2, showing that gradual updates are more robust than elimination when strategies are close in performance.
  • Damping sensitivity (eWoN; Appendix I.7): Performance and allocations are unchanged for λ from 0.01 to 0.5, so the default requires no tuning.
  • Influence vs. retraining on all setups (tMN7; Appendix I.3): The two rewards agree well on the biological tasks (Spearman ρ 0.56–0.62) and less on the regression tasks, particularly Diamonds, where the strategies perform almost identically. Despite these differences, both rewards give comparable final performance everywhere (Table 15).
  • Per-run reliability (F4Mj, tMN7; in our responses): Across 350 paired runs per method, FractAL falls below Random in 19% of runs (30–41% for every other method) and is the only method never significantly worse than Random on any setup.

Novelty and positioning (eWoN, tMN7, F4Mj). FractAL combines group influence functions and online mirror descent to assign each strategy a cheap, performance-based reward every round, without retraining, and turn these rewards into budget shares. This makes strategy selection an approximately full-information problem rather than a bandit one, so the allocation tracks the empirical performance of the strategies faithfully (Table 4) at little computational cost (Fig 4), for any differentiable loss, including the regression and perturbation tasks that motivate the paper. In relation to each reviewer's objection:

  • ALBL (tMN7) mixes strategies across sequential single-point queries. In the batch setting, it observes one scalar reward per batch, which cannot be credited to the strategies inside it.
  • TAILOR (F4Mj) mixes strategies within a batch and credits each one, but its reward measures the class balance of the acquired labels rather than model improvement. It applies only to classification and does not identify the strongest strategy in our new diagnostics (Appendix E).
  • Established building blocks (eWoN): we agree that both components are established, but neither suffices alone: SCRiBLe (soft allocation with scalar feedback) trails FractAL on all seven setups, and influence rewards with hard allocation underperform on five of seven (Table 18).

We agree with tMN7 and F4Mj that our framing overstated novelty, since mixing strategies within a batch is not itself new. We have added TAILOR to the background (Section 2.1) and Table 1, and will soften the corresponding claims in the introduction in the camera-ready version.


We hope these new results and our responses address the reviewers' concerns and clarify FractAL's contribution and scope. We thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.