acceptodds
Under review as a conference paper at ICLR 2027

Cost-aware Active Local Learning

Abstract

Active learning (AL) reduces labeling effort by selecting informative queries, but the labeling costs usually vary substantially in real applications. Existing cost-aware active learning methods usually skip costly points, discount acquisition scores by cost, or query from a cheaper but noisy oracle. We study a complementary setting in which the label of an expensive point can be estimated from labels of cheaper points. We formalize this setting as cost-aware active local learning and develop two solutions. After a base learner selects a target, the Lipschitz solution compares the cost of local labels with a direct query, while the disagreement solution queries cheap inputs until the surviving hypotheses agree on its response. For disagreement search, our theory characterizes which labels suffice and bounds total expected cost under realizability and a condition relating agreement on candidates to accuracy at the selected point, including unsuccessful exploration and direct fallback. We conducted experiments on 18 molecular datasets. The results show that adding the proposed method to existing AL methods improved molecular predictions over the evaluated labeling budgets in 289 of 351 comparisons (82%) against the same methods without adding our approach.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Active learning, learning theory, cost-aware active learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_D6aLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates active learning in settings where the labels of expensive data points can be estimated using labels from cheaper points. The authors formalize this setting as cost-aware active local learning (CALL) and propose two algorithms, CALL-LIP and CALL-DIS, together with theoretical analyses of prediction accuracy and query costs. The problem setting is interesting and practically meaningful.

Critical strengths
  • The problem formulation is practically relevant.
  • The paper provides theoretical analyses of prediction accuracy and query costs.
  • The experimental results are reported transparently, including negative findings and limitations.
Critical weaknesses
  • Limited applicability of the theoretical guarantees. Some theoretical results rely on restrictive conditions or specific algorithmic rules. For example, the cost guarantee in Eq. (5) of Theorem 1 follows from the explicit cost comparison Ct=min⁡Dt,c(xt)C_t=\min\\{D_t,c(x_t)\\}, rather than holding for the general cost formulation in Eq. (1). In particular, this rule avoids unsuccessful surrogate exploration followed by direct labeling, whereas such additional costs may arise in CALL-DIS. The applicability and practical significance of these guarantees need further clarification.

  • Insufficient analysis of the experimental findings. The central motivation of this work is to estimate expensive labels using cheaper ones. However, the ablation results in Appendix D.5.3 do not demonstrate that adding estimated labels improves predictive performance. Instead, the mean AULC decreases on both evaluated datasets. Although the authors acknowledge this limitation, the underlying reasons and implications for the proposed framework are not sufficiently investigated.

Questions for authors
  • Could the authors further clarify the applicability of the conditions underlying the theoretical guarantees? In particular, the cost bound in Eq. (5) of Theorem 1 relies on a specific cost-comparison rule. How does this guarantee relate to the general cost formulation in Eq. (1), especially when unsuccessful surrogate queries incur additional costs before direct labeling? It would also be helpful to clarify when the sufficient conditions for expected cost savings in Theorem 3 can be satisfied in practice.

  • The ablation experiments in Appendix D.5.3 show that adding estimated labels to a fixed set of purchased labels does not improve mean AULC on the two evaluated datasets. What explains this observation? Is it mainly due to estimation errors, experimental settings, parameter choices, or limitations of the proposed algorithm? A more detailed investigation would help clarify the actual contribution of the estimated labels to the overall performance.

Reviewer_qwy3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies active learning when different samples have different labeling costs. The authors propose Cost-Aware Active Local Learning (CALL), which tries to estimate the labels of expensive samples by querying cheaper related samples instead of labeling them directly. They introduce two methods: CALL-LIP, which uses the Lipschitz assumption, and CALL-DIS, which uses disagreement among models to select cheap samples. If the target label cannot be estimated, the method queries it directly. The paper also provides theoretical results on prediction accuracy and query cost under certain assumptions. Experiments on 18 molecular datasets show that CALL improves the mean AULC in 289 out of 351 comparisons (82%).

Critical strengths
  1. The idea of using cheaper samples to estimate expensive labels is interesting and can be applied to different active learning methods.
  2. The paper provides both theoretical analysis and experiments on 18 molecular datasets, with improved mean AULC in 82% of comparisons.
Critical weaknesses

1. Gap Between Theory and Practice

The theoretical guarantees of CALL-DIS rely on strong assumptions, such as the true function being included in the hypothesis space and surrogate labels being able to determine the target label. However, the ExtraTrees committee used in practice does not guarantee these assumptions. The paper also does not explain whether Morgan fingerprints can support reliable target-label estimation. Therefore, the connection between the theoretical results and the practical method could be better established.

2. Limited Reliability of Estimated Labels

The quality of the estimated labels remains a concern. In the experiments, 94.2% of accepted estimates have only one surviving tree, meaning that agreement is automatically satisfied rather than supported by multiple models. Also, 59.1% of regression estimates have errors larger than 0.5 pAffinity units. These results suggest that the current agreement-based rule may not be reliable enough for label estimation.

3. Limited Evidence for the Core Method

Although CALL-DIS improves performance in many settings, it is still unclear what mainly drives these gains. The ablation studies on two tasks show that using only purchased labels still brings improvements, while adding estimated labels slightly reduces mean AULC. Moreover, uniformly querying cheap samples performs similarly to disagreement-based querying. More experiments would help clarify whether the proposed disagreement mechanism provides benefits beyond simply collecting more cheap labels.

4. Limited Generalization of the Experimental Setting

The experiments use predefined labeling costs ({1,4,20,100}) based on synthetic accessibility scores rather than real experimental costs. Most samples are also relatively cheap, and the datasets include expensive targets with similar, low-cost candidates. In addition, targets costing 100 are queried directly instead of being estimated. It would be helpful to evaluate the method under more realistic cost settings and with fewer available cheap surrogates.

Questions for authors

See "Weaknesses".

Reviewer_TxnvOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a setting where an expensive point's label can be estimated from cheaper related points. The authors formalized the problem as cost-aware active local learning (CALL) and propose two methods: a Lipschitz branch (CALL-LIP) and a disagreement-search branch (CALL-DIS). It gives accuracy and query-cost guarantees under realizability, a query-cost lower bound, and experiments on 18 molecular datasets.

Critical strengths

The problem is important with strong motivation from drug discovery. The technical contributions include the cost-aware disagreement coefficient, the sharpened per-prediction label bound, and the cost-weighted pair-separation lower bound.

Critical weaknesses

One of my main concerns is the contribution. The framework combines mostly existing machinery, such as CALL-LIP from active local learner, of Backurs et al. (2020), CALL-DIS from standard disagreement-based Active learning.

The CALL-LIP synthetic checks are small.

Most comparison results are ties (290/351 at cost 500).

Questions for authors
  1. What evidence supports label estimation itself as the source of improvement?
  2. Can the synthetic CALL-LIP checks be scaled up to actually test the O(η⁻³) sample-size claim rather than run below the required sizes?
  3. Given ~94% of accepted estimates have one surviving tree, does the agreement test contribute anything beyond the uniform cheap-query control?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.