acceptodds
Under review as a conference paper at ICLR 2027

Defining and Benchmarking Active Data Labeling

Abstract

The availability of LLMs and VLMs enables the study of a new kind of machine learning setup, which we call Active Data Labeling (ADL): given a fixed set of data points, a novel classification task, and an oracle providing labels to this task, the goal is to annotate the entire set with oracle quality, but using as few calls to the oracle as possible. ADL is related to Active Learning, but also substantially differs from it: changing the goal from creating a model to annotating a fixed data set gives rise to a new set of affordances, methods and tradeoffs that could be explored (e.g., deferring certain classes of hard samples to the oracle while focusing on the easier, manageable ones). The ADL setup arises naturally with LLMs: a user asks to classify a set of documents or records by applying an LLM prompt to each one, and our goal is to provide a high quality annotation while minimizing actual LLM calls. We formalize the ADL learning setup, propose standardized evaluation metrics, and provide TeLa-Bench, a benchmark of cases on which it could be studied. We also present baseline algorithms that establish a lower bound on what can be achieved in the black-box ADL setup: a reduction of 43% in LLM calls while maintaining 98% agreement with the oracle.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Active Data Labeling, Active Learning, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_23LKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a new problem setting, Active Data Labeling (ADL), in which the goal is to label an entire fixed data pool with oracle-level quality while making as few oracle calls as possible. It formalizes the problem with batch/streaming and black-box/white-box variants, and proposes metrics that capture both annotation quality and computational efficiency. The authors also release TeLa-Bench, a benchmark of 30 text-classification datasets, and evaluate four baselines on it.

Critical strengths
  1. The problem is practically relevant. Labeling an entire dataset with a prompted LLM is now a common workflow, and reducing the number of expensive LLM calls without sacrificing label quality has clear real-world value.
  2. The proposed evaluation considers both annotation quality and computational efficiency. In particular, the FLOPs-based cost reduction puts the oracle’s cost and each baseline’s own computation on a common scale, so different baselines can be compared under a unified framework.
Critical weaknesses
  1. ADL is closely related to several lines of work, yet the paper neither engages with them nor explains how ADL differs.
  • Works addressing essentially the same problem: Online Cascade Learning (Nie et al., 2024), Cache & Distil (Ramírez et al., 2024), and threshold-based auto-labeling (Vishwakarma et al., 2023).

  • Works sharing key components but differing in objective or in who the expensive labeler is:

    • LLM-as-annotator active learning: LLMaAA (Zhang et al., 2023), FreeAL (Xiao et al., 2023), NoiseAL (Yuan et al., 2024), Kholodna et al. (2024), Qi et al. (2026); see also the survey by Xia et al. (2025);
    • cost-aware multi-LLM annotation: CaMVo (Elumar et al., 2025);
    • prompted weak supervision: Smith et al. (2024);
    • human–LLM co-annotation: CoAnnotating (Li et al., 2023).

    A paper introducing a new problem setting should provide a systematic taxonomy, For example, a table could compare prior work along the following dimensions: (i) the expensive labeler (human vs. LLM) and (ii) the end goal (a trained model vs. labels for a fixed pool). Such a table would make explicit what existing work covers, what it leaves open, and where ADL fits. Without it, the claim that ADL opens a "rich, unexplored research space" is not substantiated.

  1. The paper does not specify what the fully labeled pool is used for, and this determines whether full labeling is needed at all.
  • If the goal is aggregate statistics, labeling a random sample, optionally combined with prediction-powered inference, is far cheaper.
  • If the goal is a deployable model, the problem reduces to AL or distillation.
  • Record-level use cases that genuinely require every item to be labeled are not discussed.
  1. The paper reports cost only in FLOPs and does not report running time. Since LLM API calls can be parallelized or batched while the ADL baselines iterate sequentially, ADL may take longer end to end despite fewer oracle calls. Wall-clock time and hardware requirements, including threshold search, should be reported.
  2. Limited benchmarking scope. Methods that address the same problem (Online Cascade Learning, Cache & Distil, TBAL) and simple alternatives (a cheaper-LLM cascade, batch prompting) are missing. Results are averaged over 30 tasks without per-task results, variance, or ablations, and the streaming and white-box settings are never evaluated.
  3. The oracle is a single, unvalidated LLM. Neither GPT-5-mini’s agreement with gold labels nor its self-consistency across repeated calls is reported, so it is unclear whether a 98% agreement target is meaningful.
  4. Macro recall averages only over predicted classes, so ignoring a rare class raises the score. CADLC_{ADL} is undefined, and the estimation of GPT-5-mini’s FLOPs is not explained.

[1] Nie et al. Online Cascade Learning for Efficient Inference over Streams. ICML 2024.

[2] Ramírez et al. Cache & Distil: Optimising API Calls to Large Language Models. Findings of ACL 2024.

[3] Vishwakarma et al. Promises and Pitfalls of Threshold-based Auto-labeling. NeurIPS 2023.

[4] Zhang et al. LLMaAA: Making Large Language Models as Active Annotators. Findings of EMNLP 2023.

[5] Xiao et al. FreeAL: Towards Human-Free Active Learning in the Era of Large Language Models. EMNLP 2023.

[6] Yuan et al. Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLMs-Powered Assistance. ACL 2024.

[7] Kholodna et al. LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languages. ECML-PKDD 2024.

[8] Qi et al. Next Generation Active Learning: Mixture of LLMs in the Loop. AAAI 2026.

[9] Xia et al. From Selection to Generation: A Survey of LLM-based Active Learning. ACL 2025.

[10] Elumar et al. Cost-aware LLM-based Online Dataset Annotation. NeurIPS 2025.

[11] Smith et al. Language Models in the Loop: Incorporating Prompting into Weak Supervision. ACM/IMS Journal of Data Science, 2024.

[12] Li et al. CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation. EMNLP 2023.

Questions for authors

Please refer to the weakness.

Reviewer_Tf5QOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Active Data Labeling (ADL): given a fixed unlabeled collection and an expensive oracle, the objective is to label the current collection with oracle-level quality while minimizing oracle calls. The paper distinguishes batch/streaming and black-box/white-box settings, proposes quality and cost metrics, and introduces TeLa-Bench with 30 text-classification tasks and approximately 1.0164 million examples labeled by GPT-5-mini. Four baselines combine fixed sampling, active learning, confidence routing, and label propagation. At the strictest reported operating point, B4 achieves 98.6% agreement while avoiding 43.3% of oracle calls, although its estimated FLOPs reduction is negative.

Critical strengths
  • The paper identifies a common LLM batch-labeling workflow and clearly separates labeling the current collection from learning a model for future generalization.
  • TeLa-Bench has meaningful diversity in scale, domain, input length, and number of classes.
  • The four baselines form a reasonable starting point, and Section 5.7 recognizes the genuine difficulty of calibration and target-quality guarantees.
Critical weaknesses

I have serveal concerns:

  1. The reported quality operating points appear to require retrospective access to evaluation oracle labels. Section 5.7 states that a fixed confidence threshold cannot predictably control final accuracy and that multiple thresholds must be evaluated empirically to reach a desired target. If the 92/95/98% rows in Table 1 select, for each evaluation task, the threshold that satisfies the target after inspecting all oracle labels, then the reported values are oracle-aware points on a post-hoc Pareto envelope. Such a protocol uses precisely the labels that ADL is intended to avoid querying and does not demonstrate deployable target-quality control on a new task. The paper needs a label-isolated threshold-selection protocol, such as a globally fixed threshold selected on disjoint tasks, nested task-level validation, or an online risk-control method with finite-sample guarantees.
  2. Final system accuracy obscures the error rate of the automatically labeled subset. Let e denote effectiveness and a_auto the accuracy on examples not sent to the oracle. Because oracle-routed examples are correct by definition, final accuracy satisfies A_final = (1-e) + e*a_auto. Thus, increasing fallback alone can make the reported accuracy arbitrarily high. For B4 at the strictest operating point, A_final=98.6% and e=43.3% imply a_auto approximately 96.8%, not 98.6%. The current metric therefore conflates model quality with the decision to defer more examples. Coverage/selective-risk curves, conditional auto-label accuracy, and AURC are necessary to interpret the result.
  3. The headline reduction in oracle calls is not an end-to-end cost reduction. The B4 result highlighted in the abstract avoids 43.3% of oracle calls at 98.6% final accuracy, but its reported FLOPs reduction is -16.1%; under the paper's own estimate, the complete ADL pipeline is more expensive than querying the oracle on every example. The abstract and conclusions should distinguish effectiveness from actual cost reduction and should not use B4 as evidence of overall computational savings.
  4. Averaging over 30 tasks hides whether the quality constraint is satisfied on individual tasks. Table 1 reports only macro-averaged accuracy, effectiveness, recall, and FLOPs reduction. A mean accuracy above 98% can coexist with severe failures on some tasks, which is unacceptable for a method framed around strict quality guarantees. The paper does not report per-task results, the fraction of tasks satisfying each target, worst-case performance, seed variation, or uncertainty intervals.
  5. The empirical comparison does not establish that ADL is distinct from, or better served than, existing selective prediction frameworks. The paper compares only four internally designed baselines. It does not include selective classification/reject-option methods, conformal or risk-controlling prediction, learning to defer, adaptive cascades, Online Cascade Learning, or simple embedding-based kNN/deduplication. Without these comparisons, it remains unclear whether ADL requires a new paradigm or is a particular combination of transductive active learning and selective prediction.

and I also several comments:

  • The macro-recall definition may fail to penalize completely missed classes. The paper defines L as the set of classes assigned by the labeler. If a rare class is never predicted, it may not enter the average at all, contradicting the stated motivation for macro recall. The denominator should use the complete oracle label space. The accuracy paragraph also defines the fraction of labels that differ from the oracle, which is an error rate rather than accuracy.
  • The FLOPs accounting is not reproducible. GPT-5-mini is proprietary, yet the paper does not provide the architecture assumptions or formula used for per-example oracle FLOPs. It is also unclear whether C_ADL includes all ModernBERT training passes, embedding extraction, K-Means, KNN graph construction, label propagation, calibration, repeated retraining, multiple thresholds evaluated in Section 5.7, and failed/retried oracle generations.
  • The prose and pseudocode disagree about B4 acquisition. Section 5.4 describes density-weighted uncertainty combining entropy with graph centrality, whereas Algorithm 5 selects argmax Entropy(f(x)) and contains no centrality term. This changes the acquisition rule rather than merely omitting an implementation detail.
  • Algorithms 2-5 contain unresolved state and naming errors. Algorithm 2 calls TrainRouter, while the auxiliary function is named TrainCalibratedRouter; Algorithm 1 refers to undefined D_all; and D_auto is not initialized in Algorithms 3-5. In Algorithm 3, D_auto is also undefined if the stream ends before the confidence threshold is reached.
  • The calibration pool may be contaminated by pseudo-labels. Algorithms 4-5 add D_auto to D_labeled and subsequently split D_labeled into training and calibration sets. Read literally, automatically assigned labels can therefore be treated as ground truth when calibrating whether predictions match the oracle. If the implementation maintains a separate oracle-only pool, the state variables and pseudocode should say so explicitly.
  • B2 repeatedly uses the same held-out set as a stopping criterion. Evaluating the same D_test after every training round adaptively overfits the stopping decision to that set. The paper should clarify whether fresh calibration data, confidence sequences, or any correction for repeated looks is used.
  • The benchmark only partially matches the motivating use case. Many tasks are classic public datasets such as AG News, SST-2, SNLI, and Amazon reviews, whose taxonomies and examples may already be familiar to GPT-5-mini and ModernBERT. This setting does not directly test rapid adaptation to genuinely novel, organization-specific labeling instructions. A temporal holdout, newly constructed taxonomy, or real enterprise task would better test the central claim.
  • Exact dataset identifiers and revisions, split construction, sampling when capping tasks at 50k examples, deduplication, licenses, GPT-5-mini snapshot and decoding settings, retry policy, and the complete task prompts are not available in the submission.
Questions for authors

Questions for the Authors

  1. How exactly is the threshold for each quality level in Table 1 selected, and which oracle labels are available during that selection?
  2. Must the target accuracy be satisfied independently on every task, or only after averaging across 30 tasks? Please report the number of tasks satisfying each constraint, the worst-task result, and seed-level uncertainty.
  3. Please report effectiveness together with the conditional accuracy/error rate of the automatically labeled subset and full risk-coverage curves.
  4. Please provide the complete FLOPs formula and a component-wise cost table. Why does a 43.3% reduction in calls produce -16.1% FLOPs reduction?
  5. In the actual implementation, are oracle labels, calibration labels, and pseudo-labels maintained in separate pools? Which acquisition rule was used for B4: entropy only or density-weighted uncertainty?
  6. Why are selective classification, conformal/risk-controlling prediction, learning to defer, adaptive cascades, and Online Cascade Learning absent from the comparison?
  7. Can you release the complete per-task table and add tasks with genuinely novel taxonomies that are unlikely to have appeared in pretraining?
Reviewer_pxGxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a novel task, active data labeling, which, given an unlabeled dataset and a cost-intensive oracle tries to query the oracle systematically for labels such that some semi-supervised pseudo-labeling scheme will be overall successful in assigning labels to the full dataset. Based on the language domain, the authors also introduce TeLa-Bench, a composite dataset of text bodies and GPT-5-mini-queried oracle responses. Apart from the definition of the new task, the authors present different flavors akin to similar active learning flavors (e.g., pool- vs. stream-based). Evaluation metrics are defined which measure different aspects of ADL methods like model recall, fraction of oracle-queries at certain accuracy or cost reduction and an array of 4 baseline models presented and evaluated experimentally.

Critical strengths

S1: The proposed new task seems interesting and of considerable industrial use and development of ADL methods could save considerable amounts of effort/energy. S2: The general strategy of the benchmark construction and the selected baselines seem like interesting contributions.

Critical weaknesses

W1: The experimental results and their discussion are holding back the contributions: The obtained results in Table 1 deserve much clearer discussion and interpretation. Currently, this happens within the 6 lines following l.427 which is insufficient -> Q1, Q2 Generally, the implementation details should (at least in supplementary material) document hyperparameters of the models -> Q3, Q4. Finally, careful ablation studies per baseline should be conducted to validate them, e.g., the sample size/fraction used in B1.

W2: Overall, the manuscript seems in parts unfinished and unrefined. The ideas from section 3.2 are nowhere made explicit experimentally. The cost-reduction metric could have been explored more (e.g., via input length - larger input/context = higher cost). Some sections are repetitive (ll. 383) and the paper contains numerous typos or grammatical errors (e.g., l.265 "Tele-Bench") and the benchmark acronym TeLa-Bench is nowhere explained.

Questions for authors

Q1: Could the authors discuss the baseline rankings in Table 1? I would learn from this table that B1, the most naive baseline, is a very solid tradeoff and too sophisticated methods like B4 run the danger of being too computationally expensive. Can you put the results into perspective?

Q2: The authors state that they execute 3 runs (l.427) - why are no variances reported in table 1? Are there systematic differences in the metric variance of the baselines?

Q3: What is the utilized architecture of the B1/3 MLP? What "representation" (l.422) is used and why?

Q4: What are the dataset split sizes or the hold-out fractions? Would it not make sense to isolate residual pseudo-labeling performance of the model on a completely separate holdout test set which was never accessible?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.