acceptodds
Under review as a conference paper at ICLR 2027

Zero-pool Active Learning: Choosing Experiments Through Metadata Alone

Abstract

When datasets are built through prospective generation, whether via physical experiments or complex simulations, the primary cost lies in producing the samples rather than annotating an existing pool. Active learning has the right objective for this setting but the wrong assumptions: it presumes an initial labeled set, a pool of unlabeled candidates to score, and a reliable post hoc oracle, none of which can be assumed when data must be generated. We formalize this zero-pool setting and introduce ZALMBIES, an algorithm that acquires data by specifying experimental conditions rather than by selecting instances. Because such experiments are defined by a small number of categorical choices, we place a hierarchical Bayesian ANOVA model over the metadata lattice, fit it to the primary model's realized out-of-fold failure, and interpolate between additive and multiplicative geometry with a Box-Cox power selected online. Candidate bins are scored by weighing predicted failure against an effective sample size that captures the statistical strength already borrowed from the surrounding lattice, and batches are built greedily with each selection depressing the scores of overlapping bins. We show that acquisitions are -optimal at cold start for the underlying linear model, and that under our exploration setting the resulting allocation asymptotically converges to Neyman proportions. Across four benchmarks spanning glucose simulation, quadruped robotics, wearable sensing, and sleep staging, ZALMBIES reaches the performance of random sampling using roughly half the budget. A sample here is an experiment, not an annotation, so the budget is spent in months of collection and the saving is one of schedule as much as of cost.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
active learning, experimental design, optimal design, data-efficient learning, time series, Bayesian linear models, zero-pool active learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LJEpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates active learning in prospective data-collection settings, where the primary cost arises from generating new samples rather than annotating an existing unlabeled pool. The authors formalize this setting as zero-pool active learning and propose ZALMBIES, a metadata-based acquisition method that selects experimental conditions without requiring access to candidate samples. The problem is practically meaningful, and the proposed framework presents an interesting integration of active learning and optimal experimental design.

Critical strengths
  • A practically meaningful problem formulation.

The paper addresses a relevant active learning setting in which data acquisition requires conducting experiments rather than labeling existing samples. This formulation is motivated by practical applications and provides an interesting perspective on integrating active learning with experimental design.

Critical weaknesses
  • Limited scalability.

The full high-order ANOVA encoding has an exponentially growing feature dimension M=∏d=1D(md+1)M=\prod_{d=1}^{D}(m_d+1). Maintaining the corresponding information matrix may lead to substantial computational and memory costs, raising concerns about scalability to high-dimensional metadata spaces.

  • Unclear necessity of the Box–Cox transformation.

The authors introduce Box–Cox to capture multiplicative and skewed interactions. However, the full ANOVA expansion can already represent arbitrary functions over categorical combinations, including multiplicative and nonlinear interactions. Thus, Box–Cox does not increase the model's expressive capacity, and its additional benefits remain unclear.

  • Limited relevance of some theoretical results to the proposed framework.

The paper presents extensive theoretical analyses, but their connection to the actual algorithm is sometimes unclear. For example, the submodularity analysis establishes a 1−1/e1-1/e-approximation guarantee only for the log-determinant objective under constant failure weights, rather than for the actual ZALMBIES acquisition rule or downstream learning objective. While these results explain certain special cases, their direct theoretical support for the complete framework is limited. I suggest focusing the theoretical presentation on results more directly relevant to the proposed algorithm and removing less essential discussions.

Questions for authors
  • Scalability.

Could the authors provide a detailed complexity analysis and scalability experiments? Could restricting the interaction order or using structured approximations improve scalability, and how would these changes affect the algorithm's performance and theoretical properties?

  • Box–Cox transformation.

Could the authors clarify the essential contribution of Box–Cox? Given that the full ANOVA expansion already captures arbitrary categorical interactions, what additional theoretical or empirical benefits does this transformation provide? Without clear evidence of its benefits, this component appears potentially redundant.

Reviewer_Y8HkOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In cases where the cost of a dataset is primarily in generating the data, it’s helpful to know which data should be generated/acquired. The paper describes the zero-pool active learning setting, which uses metadata (which gives a picture of experimental conditions) to determine which data will most improve the predictive performance. ZALMBIES, the proposed method, first uses hierarchical ANOVA decomposition over metadata categories, fit it to the predicted failure, apply a Box-Cox transformation to account for a mix of additive/multiplicative effects of model failures. The acquisition strategy greedily uses predictive failure and effective sample size signals, and the authors demonstrate that ZALMBIES improves over random sampling across four benchmarks datasets (two simulated, two real).

Critical strengths
  • The authors put forth a new problem framing that is complimentary to existing literature on optimal experimental design and active labeling, but novel. Their description of the problem setting and justification for where this is applicable in real-world applications is clear and well-presented.
  • The paper’s evaluation is robust. The main paper experiments span two simulated datasets and two real-world datasets, and include ablations of the free parameters in the appendix, and the extension (with exhaustive baselines) to pool-based AL in the appendix.
  • Overall, the presentation of the results is clear, and the paper is well-written. The authors qualify their claims, and discuss limitations where needed (I particularly appreciated the discussion of the suitability of the Sleep-EDF dataset in the appendix, and the assumptions for Theorems 3-4 that may not hold with real data).
Critical weaknesses
  • The authors briefly describe the datasets in the main text, but our understanding of the results could benefit from a clear, precise definition of exactly how the metrics are calculated for each benchmark dataset, and exactly what the target for each dataset is.
  • The authors carefully explain the need for a Box-Cox transformation, but not the use of the hierarchical ANOVA decomposition for their problem setting.
  • While the improvement over the random baseline is fairly obvious, the margin over the adapted LinUCB method is less so. It looks as though their error ranges are fairly overlapped (except on night-level sampling in Appendix Figure 3). While this not the goal in this paper per L349-351, how do we interpret this comparison across the benchmarks?
  • It would be helpful to see the pure exploitation variant of ZALMBIES as an additional baseline. Why does Include citations such as [1] for the choice in the Beta parameter (around L291 or L303).

[1] Doucet, P., Estermann, B., Aczel, T., & Wattenhofer, R. (2024). Bridging diversity and uncertainty in active learning with self-supervised pre-training. arXiv preprint arXiv:2403.03728.

Questions for authors
  • Could you justify your use of ANOVA decomposition more deliberately? It seems like the most appropriate method for your problem-setting, but it should be justified in the main text.
  • How does high exploration (Beta = 2) compare to the metadata equal baseline? How does the pure exploitation (Beta = 0) compare with the proposed method?
  • How does convergence behavior change with batch size? Consider adding this ablation to the appendix.
  • The authors discuss rare categories as a condition that may break the symmetry requirement in Theorem 4. Do the authors have suggestions for what might change about the method to account for the informativeness of rare categories?
Reviewer_aME5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies zero-pool active learning, where acquisition specifies experimental conditions rather than selecting instances from an unlabeled pool. It introduces ZALMBIES, which fits a hierarchical Bayesian ANOVA model over the metadata lattice to the primary model’s realized out-of-fold failure. A Box–Cox power is selected online, and candidate bins are scored by predicted failure divided by an effective sample size, with sequential updates for batch selection. The paper provides D-optimality results at cold start for the underlying linear model and a limiting allocation related to Neyman allocation under fixed weights.

Critical strengths

The problem setting is practically relevant and is close to active experimental design. In many applications, the expensive part is running an experiment or collecting data, and the decision is which experimental conditions to use next. I find this closer to practical data collection needs than selecting samples from an existing unlabeled pool. Using metadata available before an experiment to guide acquisition is a useful direction, and the results across simulated and real datasets provide some evidence that this can reduce the required collection budget.

Critical weaknesses
  1. The problem setting is practically relevant and is close to active experimental design. In many applications, the expensive part is running an experiment or collecting data, and the decision is which experimental conditions to use next. I find this closer to practical data collection needs than selecting samples from an existing unlabeled pool. Using metadata available before an experiment to guide acquisition is a useful direction, and the results across simulated and real datasets provide some evidence that this can reduce the required collection budget.
  2. High out-of-fold error can result from insufficient data, but also from label noise or limitations of the primary model. The effective sample size penalty discourages repeatedly sampling the same region, but does not distinguish these causes. In particular, the allocation limit can still assign a large budget share to bins with persistently high, irreducible error.
Questions for authors
  1. Discrete experimental design without access to model gradients has already been studied, for example, in “Optimizing Sequential Experimental Design with Deep Reinforcement Learning” (Blau et al., ICML 2022). The Related Work claim that Deep BOED requires continuous, differentiable spaces therefore seems too broad.
  2. Using the stated thresholds and the official subject metadata, the combination female, age <28, and weight-to-height ratio >0.45 has no participants. Several other demographic combinations have only one participant. How does this agree with the statement that all bins contain more than one participant?
  3. Using the stated thresholds and the official subject metadata, the combination female, age <28, and weight-to-height ratio >0.45 has no participants. Several other demographic combinations have only one participant. How does this agree with the statement that all bins contain more than one participant?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.