- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates active learning in settings where the labels of expensive data points can be estimated using labels from cheaper points. The authors formalize this setting as cost-aware active local learning (CALL) and propose two algorithms, CALL-LIP and CALL-DIS, together with theoretical analyses of prediction accuracy and query costs. The problem setting is interesting and practically meaningful.
Critical strengths
- The problem formulation is practically relevant.
- The paper provides theoretical analyses of prediction accuracy and query costs.
- The experimental results are reported transparently, including negative findings and limitations.
Critical weaknesses
-
Limited applicability of the theoretical guarantees. Some theoretical results rely on restrictive conditions or specific algorithmic rules. For example, the cost guarantee in Eq. (5) of Theorem 1 follows from the explicit cost comparison , rather than holding for the general cost formulation in Eq. (1). In particular, this rule avoids unsuccessful surrogate exploration followed by direct labeling, whereas such additional costs may arise in CALL-DIS. The applicability and practical significance of these guarantees need further clarification.
-
Insufficient analysis of the experimental findings. The central motivation of this work is to estimate expensive labels using cheaper ones. However, the ablation results in Appendix D.5.3 do not demonstrate that adding estimated labels improves predictive performance. Instead, the mean AULC decreases on both evaluated datasets. Although the authors acknowledge this limitation, the underlying reasons and implications for the proposed framework are not sufficiently investigated.
Questions for authors
-
Could the authors further clarify the applicability of the conditions underlying the theoretical guarantees? In particular, the cost bound in Eq. (5) of Theorem 1 relies on a specific cost-comparison rule. How does this guarantee relate to the general cost formulation in Eq. (1), especially when unsuccessful surrogate queries incur additional costs before direct labeling? It would also be helpful to clarify when the sufficient conditions for expected cost savings in Theorem 3 can be satisfied in practice.
-
The ablation experiments in Appendix D.5.3 show that adding estimated labels to a fixed set of purchased labels does not improve mean AULC on the two evaluated datasets. What explains this observation? Is it mainly due to estimation errors, experimental settings, parameter choices, or limitations of the proposed algorithm? A more detailed investigation would help clarify the actual contribution of the estimated labels to the overall performance.