- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates active learning in prospective data-collection settings, where the primary cost arises from generating new samples rather than annotating an existing unlabeled pool. The authors formalize this setting as zero-pool active learning and propose ZALMBIES, a metadata-based acquisition method that selects experimental conditions without requiring access to candidate samples. The problem is practically meaningful, and the proposed framework presents an interesting integration of active learning and optimal experimental design.
Critical strengths
- A practically meaningful problem formulation.
The paper addresses a relevant active learning setting in which data acquisition requires conducting experiments rather than labeling existing samples. This formulation is motivated by practical applications and provides an interesting perspective on integrating active learning with experimental design.
Critical weaknesses
- Limited scalability.
The full high-order ANOVA encoding has an exponentially growing feature dimension . Maintaining the corresponding information matrix may lead to substantial computational and memory costs, raising concerns about scalability to high-dimensional metadata spaces.
- Unclear necessity of the Box–Cox transformation.
The authors introduce Box–Cox to capture multiplicative and skewed interactions. However, the full ANOVA expansion can already represent arbitrary functions over categorical combinations, including multiplicative and nonlinear interactions. Thus, Box–Cox does not increase the model's expressive capacity, and its additional benefits remain unclear.
- Limited relevance of some theoretical results to the proposed framework.
The paper presents extensive theoretical analyses, but their connection to the actual algorithm is sometimes unclear. For example, the submodularity analysis establishes a -approximation guarantee only for the log-determinant objective under constant failure weights, rather than for the actual ZALMBIES acquisition rule or downstream learning objective. While these results explain certain special cases, their direct theoretical support for the complete framework is limited. I suggest focusing the theoretical presentation on results more directly relevant to the proposed algorithm and removing less essential discussions.
Questions for authors
- Scalability.
Could the authors provide a detailed complexity analysis and scalability experiments? Could restricting the interaction order or using structured approximations improve scalability, and how would these changes affect the algorithm's performance and theoretical properties?
- Box–Cox transformation.
Could the authors clarify the essential contribution of Box–Cox? Given that the full ANOVA expansion already captures arbitrary categorical interactions, what additional theoretical or empirical benefits does this transformation provide? Without clear evidence of its benefits, this component appears potentially redundant.