- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a new problem setting, Active Data Labeling (ADL), in which the goal is to label an entire fixed data pool with oracle-level quality while making as few oracle calls as possible. It formalizes the problem with batch/streaming and black-box/white-box variants, and proposes metrics that capture both annotation quality and computational efficiency. The authors also release TeLa-Bench, a benchmark of 30 text-classification datasets, and evaluate four baselines on it.
Critical strengths
- The problem is practically relevant. Labeling an entire dataset with a prompted LLM is now a common workflow, and reducing the number of expensive LLM calls without sacrificing label quality has clear real-world value.
- The proposed evaluation considers both annotation quality and computational efficiency. In particular, the FLOPs-based cost reduction puts the oracle’s cost and each baseline’s own computation on a common scale, so different baselines can be compared under a unified framework.
Critical weaknesses
- ADL is closely related to several lines of work, yet the paper neither engages with them nor explains how ADL differs.
-
Works addressing essentially the same problem: Online Cascade Learning (Nie et al., 2024), Cache & Distil (Ramírez et al., 2024), and threshold-based auto-labeling (Vishwakarma et al., 2023).
-
Works sharing key components but differing in objective or in who the expensive labeler is:
- LLM-as-annotator active learning: LLMaAA (Zhang et al., 2023), FreeAL (Xiao et al., 2023), NoiseAL (Yuan et al., 2024), Kholodna et al. (2024), Qi et al. (2026); see also the survey by Xia et al. (2025);
- cost-aware multi-LLM annotation: CaMVo (Elumar et al., 2025);
- prompted weak supervision: Smith et al. (2024);
- human–LLM co-annotation: CoAnnotating (Li et al., 2023).
A paper introducing a new problem setting should provide a systematic taxonomy, For example, a table could compare prior work along the following dimensions: (i) the expensive labeler (human vs. LLM) and (ii) the end goal (a trained model vs. labels for a fixed pool). Such a table would make explicit what existing work covers, what it leaves open, and where ADL fits. Without it, the claim that ADL opens a "rich, unexplored research space" is not substantiated.
- The paper does not specify what the fully labeled pool is used for, and this determines whether full labeling is needed at all.
- If the goal is aggregate statistics, labeling a random sample, optionally combined with prediction-powered inference, is far cheaper.
- If the goal is a deployable model, the problem reduces to AL or distillation.
- Record-level use cases that genuinely require every item to be labeled are not discussed.
- The paper reports cost only in FLOPs and does not report running time. Since LLM API calls can be parallelized or batched while the ADL baselines iterate sequentially, ADL may take longer end to end despite fewer oracle calls. Wall-clock time and hardware requirements, including threshold search, should be reported.
- Limited benchmarking scope. Methods that address the same problem (Online Cascade Learning, Cache & Distil, TBAL) and simple alternatives (a cheaper-LLM cascade, batch prompting) are missing. Results are averaged over 30 tasks without per-task results, variance, or ablations, and the streaming and white-box settings are never evaluated.
- The oracle is a single, unvalidated LLM. Neither GPT-5-mini’s agreement with gold labels nor its self-consistency across repeated calls is reported, so it is unclear whether a 98% agreement target is meaningful.
- Macro recall averages only over predicted classes, so ignoring a rare class raises the score. is undefined, and the estimation of GPT-5-mini’s FLOPs is not explained.
[1] Nie et al. Online Cascade Learning for Efficient Inference over Streams. ICML 2024.
[2] Ramírez et al. Cache & Distil: Optimising API Calls to Large Language Models. Findings of ACL 2024.
[3] Vishwakarma et al. Promises and Pitfalls of Threshold-based Auto-labeling. NeurIPS 2023.
[4] Zhang et al. LLMaAA: Making Large Language Models as Active Annotators. Findings of EMNLP 2023.
[5] Xiao et al. FreeAL: Towards Human-Free Active Learning in the Era of Large Language Models. EMNLP 2023.
[6] Yuan et al. Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLMs-Powered Assistance. ACL 2024.
[7] Kholodna et al. LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languages. ECML-PKDD 2024.
[8] Qi et al. Next Generation Active Learning: Mixture of LLMs in the Loop. AAAI 2026.
[9] Xia et al. From Selection to Generation: A Survey of LLM-based Active Learning. ACL 2025.
[10] Elumar et al. Cost-aware LLM-based Online Dataset Annotation. NeurIPS 2025.
[11] Smith et al. Language Models in the Loop: Incorporating Prompting into Weak Supervision. ACM/IMS Journal of Data Science, 2024.
[12] Li et al. CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation. EMNLP 2023.
Questions for authors
Please refer to the weakness.