acceptodds
Under review as a conference paper at ICLR 2027

AdaLabel: Learning Signed Annotation Value for Budgeted Data Annotation

Abstract

Stronger annotators can improve labels on average without being correct on every example. Replacing a cheap label can rescue an error or regress a correct annotation. We therefore define signed annotation value (SAV) as the expected change in annotation correctness caused by an upgrade, conditional only on information available before the strong call. AdaLabel learns the CC/WC/WW/CW outcome distribution from paired Meta supervision and ranks upgrades by . In an independently and prospectively frozen Yahoo Answers evaluation, AdaLabel improves over the prespecified Direct MLP signed-value comparator by 73–170 net corrections under matched logical-token budgets, while remaining competitive with strong heuristic alternatives. An exploratory frozen-record replay recovers WC-only under constant predicted regression risk and produces dataset-dependent ranking divergence as heterogeneity is restored, without guaranteeing utility gains. Exploratory frozen-record attribution supports instance-level value learning beyond class-only priors, and independent Amazon results show that retaining four outcomes improves joint-failure auditability. Results vary across budgets and datasets, so they do not establish universal dominance.

Primary area:
datasets and benchmarks
Keywords:
LLM annotation, data annotation, active learning, budgeted annotation, selective labeling, label correction, weak supervision, uncertainty estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_xkzuOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which existing cheap annotations should be replaced by a stronger, imperfect annotator under a limited budget. It defines signed annotation value as the conditional expected improvement in label correctness, accounting for both corrections and regressions. AdaLabel estimates four joint correctness outcomes from paired reference-labeled Meta data and ranks target examples using only information available before the strong annotation.

Critical strengths
  1. This work introduces a well-defined, practically relevant objective with an explicit information boundary. The method distinguishes correctable errors from joint failures and explicitly penalizes harmful replacements. Learning from paired Meta outcomes while restricting deployment to cheap-side information gives a concrete implementation of prospective annotation allocation. This is more informative than treating stronger annotations as uniformly beneficial.
  2. Yahoo rankings are reported as frozen before target strong outcomes were accessed, with resource matching and paired uncertainty estimates. Improvements of 73–170 net corrections over Direct MLP are meaningful within that comparison. The authors also disclose the post-Gold status of DBPedia14, weak or negative comparisons, and limited downstream significance. This transparency strengthens confidence in the narrower empirical conclusions.
Critical weaknesses
  1. The incremental allocation benefit of the distinguishing components is insufficiently established. In the prospective Yahoo evaluation, AdaLabel improves over WC-only by 0, −4, −2, and +3 net corrections; on 20 Newsgroups, the differences are +4, +2, +8, and 0. Although the CW correction changes selections and avoids regressions, these benefits are largely offset by lost rescues. Furthermore, every four-state versus two-correctness interval includes zero, leaving the necessity of the four-state parameterization unestablished. The larger DBPedia advantage is drawn from a corrective post-Gold audit. Thus, the strongest evidence does not demonstrate a consequential allocation advantage attributable to the method’s distinctive components.
  2. The evaluation establishes a narrower practical advantage than the headline comparison suggests. Yahoo gains of 73–170 corrections are measured against Direct MLP, whereas gains over Direct Linear are only 13–22 and WC-only is essentially competitive; no Yahoo Cheap-error result is reported. Practical cost-effectiveness also remains unresolved because paired Meta supervision—including 2,000 reference-labeled examples—is excluded from target budgets, with no cost-inclusive or amortization analysis. The low observed annotator accuracies further motivate validation across alternative annotation interfaces. Missing discussion of modern LLM routing and active label cleaning also leaves the novelty and baseline coverage incomplete.
Questions for authors
  1. Beyond the annotation-construction application, what precise result distinguishes signed annotation value from conditional risk-difference allocation in budgeted triage? How should the two-correctness identity above affect the stated formulation contribution?
  2. Do any existing prospectively frozen comparisons show a practically meaningful, statistically supported advantage over WC-only or two-correctness estimation? Are downstream results against these alternatives available? Such evidence would materially strengthen the contribution.
  3. Appendix Q uses Qwen3-1.7B as the strong annotator for Meta supervision but Qwen3-8B for Gold. What motivates this change, and how should it affect interpretation of the auditability result under the stated stable-hierarchy assumption? Is there a matched binary WW detector to separate the value of four-state modeling from simply providing explicit WW supervision?
  4. What target-population scale is needed to amortize paired Meta annotation and reference-label costs? An accounting based on the existing experiments would help determine when the proposed allocation produces an end-to-end resource benefit.
Reviewer_4cT3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which cheap labels to replace when a stronger annotator is available on a limited budget. The stronger annotator can both fix errors and introduce new ones. The paper scores each example by the chance that relabeling fixes a wrong label (a rescue) minus the chance that it breaks a correct one (a regression). These chances are learned from a separate "Meta" set that has cheap, strong and true labels. The ranking uses only information available before the strong annotator is called. In the main Yahoo Answers experiment, the rankings were fixed before the results were seen. There, the method gives more net label corrections than a small MLP baseline chosen in advance, at the same token budget. Other experiments look at different ways to model the outcomes, how much regression risk varies, downstream training on Banking77, transfer to other datasets, and detecting cases where both annotators are wrong.

Critical strengths
  1. The method picks better examples to relabel than simple uncertainty scores such as entropy or margin. It learns when the strong model will actually fix a label, rather than only looking at where the cheap model is unsure. For example, on 20 Newsgroups at the 40% budget it gets 2,761 net corrections, compared with about 2,500 for entropy and margin.

  2. The method is simple and cheap to use. It is a small classifier trained on the cheap model's confidence scores. Ranking a large dataset costs almost nothing and never needs a call to the strong model. Any better classifier can be swapped in. The output is just a list of examples to relabel, so it fits easily into an existing labeling pipeline.

  3. The results are easy to check. The authors provide fixed rankings, outcome tables and scripts that reproduce the reported numbers. In the main Yahoo experiment, the rankings were also fixed before the results were seen, which makes the main comparison more trustworthy.

Critical weaknesses
  1. The method is not very new. Its score is just the expected accuracy gain from relabeling, P(strong is right) − P(cheap is right), which learning to defer and LLM routing methods already use. The paper says it differs because the decision is made while building training data, but labeling with an LLM is itself a prediction.

  2. The regression penalty is what makes the method different, but it does not help much. Compared with WC-only, which ignores regressions, the differences are 0, −4, −2 and +3 net corrections on Yahoo, and close to zero on 20 Newsgroups. The only large gap is on DBPedia14, in a table made after the results were seen.

  3. The main gain is over a weak baseline, and the full cost is not counted. Direct MLP is the weakest learned method in both main tables, and simple baselines such as random are missing from the Yahoo table. The method also needs a Meta set with true labels, 25–200% the size of the target set, which is left out of the budget.

Questions for authors
  1. Every example already has a noisy cheap label, and relabeled examples get a second noisy label from the strong model, which is close to the weak supervision setting. Weak supervision methods usually combine noisy labels instead of keeping only one. How would combining the cheap and strong labels compare with replacing the cheap label?

  2. The Meta set's true labels are used to train the model that picks which examples to relabel. A simpler use is to give them to the strong annotator as in-context examples, which may directly improve its accuracy. At the same budget, how does your method compare with this baseline?

  3. The only difference between your score and WC-only is the penalty for predicted regressions, yet the two are almost tied on Yahoo (0, −4, −2 and +3 net corrections) and 20 Newsgroups. What evidence shows that this penalty gives a real benefit?

Reviewer_rAR9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper explores the question of how to best utilize a strong annotator (presumably some expensive LLM) to re-label data that was previously labeled by a weaker annotator (presumably some cheap LLM). Under a fixed budget B, they propose to predict the “signed annotation value”: the gap between the probability of a “rescue” (correcting a wrong label) and a “regression” (corrupting a correct label). Given these predictions, one would then use the strong annotator to re-label the B highest-scoring instances.

As stated in the abstract, there is no empirical evidence that the proposed approach is more effective than the included baselines (specifically, WC-only). There is also no evidence in the main paper to suggest that re-labeling in the proposed manner would have any measurable impact on a downstream model.

Critical strengths
  1. The question of how to best allocate automated labeling resources is quite relevant. Thus this paper is timely and, given a compelling enough method, would be useful to the overall ML community.
  2. The proposed approach is simple, straightforward, and intuitive. Given stronger results, these characteristics would increase the odds of adoption by the broader community.
Critical weaknesses
  1. The paper does not demonstrate empirically that the proposed approach is any better than the presented baselines (such as ranking by P(WC∣scheap(x))P(WC | s_{cheap}(x)) only). In fact, results appear to show that the predicted P(CW∣scheap(x))P(CW | s_{cheap}(x)) is roughly constant, so there is little to no ranking benefit induced by this extra “signed” term. This suggests that there is insufficient information in scheap(x)s_{cheap}(x) to differentiate between data that will likely be corrupted vs. corrected when using a stronger annotator.
  2. The main paper makes no attempt at connecting the proposed technique to better downstream performance. Does re-labeling the top B instances w.r.t. SAV actually result in a meaningful difference when models are trained on the refined dataset? There is reason to suspect this bump would be minimal, especially since a simple top-B ranking does not consider diversity of samples. Either way, without experiments explicitly investigating this question, the method seems inconsequential.
  3. The theoretical results presented in the paper are either exceptionally obvious (Lemma 1, which states the well-known fact that ranking is preserved up to a constant shift) or vacuous (Proposition 1, which assumes “additive utility” without definition and then relies on it to state that, essentially, if we assume that the stronger annotator will replace an incorrect label, the number of correct labels is guaranteed to increase). These do not provide any value to the community and should be removed.
Questions for authors
  • What is the downstream impact of a high U(S)U(S)? Does re-labeling the top-B as you suggest result in a meaningful boost for models trained on this data?
  • Can you demonstrate a significant difference between AdaLabel and WC-only?
  • Were the baselines you compared against proposed by you, or by prior work? It is unclear to me where your contribution slots in against existing literature, and the paper does little to no exposition on the origin (or details) of “Direct MLP”, “Direct Linear”, and “WC-only”.
  • Can you clarify the purpose of Lemma 1 and Proposition 1 in the text?
  • Why did you defer all details of your predictive method (to predict the SAV) to the appendix? This also includes all details about your baselines and other crucial experiments.
  • Why does your paper have 17 appendices spread across 22 pages?
Reviewer_RPZDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AdaLabel addresses a practical question in budgeted data annotation: which existing cheap labels are worth replacing with labels from a stronger annotator? A stronger annotator can correct an error, but can also replace a correct label with an incorrect one. The paper defines Signed Annotation Value (SAV) as the expected gain in correctness from an upgrade, accounting for both outcomes. The method learns a four-outcome model from paired cheap, strong, and reference annotations, then ranks examples using the predicted probability of a correction minus the predicted probability of a regression. The evaluation includes Yahoo Answers, DBPedia14, Banking77, and 20 Newsgroups. In the prospectively frozen Yahoo Answers experiment, AdaLabel yields 73–170 more net corrections than Direct MLP under matched logical-token budgets. Ablations, downstream classification experiments, transfer analyses, and an independent Amazon auditability study help establish the scope of the method, although the results do not show consistent superiority over all alternatives.

Critical strengths
  1. I find the formulation useful because it treats stronger annotators as fallible. Selecting an example for another annotation is only worthwhile if the expected correction outweighs the risk of introducing an error. Making that tradeoff explicit gives the allocation objective a clear practical meaning and addresses a limitation of selection rules based only on uncertainty or cheap-label error likelihood.
  2. The evaluation also gives the proposal a reasonable empirical basis. In particular, the prospective Yahoo Answers experiment fixes the selection rankings before the stronger annotation outcomes are accessed, which makes the comparison more persuasive. The ablations and downstream experiments help distinguish the value of the formulation from the behavior of a particular estimator, and they show where the advantages are less consistent.
Critical weaknesses
  1. The benefit of explicitly modeling regression risk is less convincing than the improvement over Direct MLP alone might suggest. On Yahoo Answers, AdaLabel is very close to WC-only, and WC-only does slightly better at some budgets. Since accounting for regressions is central to the paper’s motivation, I would like clearer evidence of when that extra modeling changes allocation enough to produce a meaningful practical gain.
  2. The paired supervision needed for the Meta dataset is another concern. Training the model requires cheap annotations, strong annotations, and reference labels, yet the main selection budgets do not include the initial cost of collecting that supervision. This makes it difficult to judge whether the method is cost-effective for a new task, particularly when the available annotation budget is small.
Questions for authors

The following are clarification requests that could affect my assessment, rather than suggestions for future work.

  1. Since WC-only performs nearly as well as AdaLabel on Yahoo Answers, under what conditions does explicitly modeling regression risk provide a meaningful practical advantage?
  2. How sensitive is AdaLabel to the size and quality of the Meta dataset? Would it remain effective with substantially fewer paired examples?
  3. Can you provide an end-to-end cost comparison that includes collecting the Meta annotations, as well as the annotation budget used during selection?

Comments on the paper

2 comments

Thank the authors for the response and the revision. The submission has improved with a narrower claim. While some of my concerns are acknowledged rather than fully resolved:

  • For HateXplain, freezing the rankings before revealing the Test labels is good practice. However, WC-only here is a separately trained binary model, not P(WC) from the 4-state model as in Yahoo, and it scores below Random (−36 vs. +13.7 at 20%). Please report the 4-state model's own P(WC) ranking and two-correctness on the frozen Test records. Without these, the result supports adding a CW term but not the four-state model specifically.
  • My earlier point that no Yahoo Cheap-error result is reported was not addressed. Random also appears only as significance counts, not as utilities.
  • From the provided Yahoo numbers, the 2000 Meta strong calls spent on random target upgrades would yield about 596 net corrections. AdaLabel's advantage over random at 40% is about 226. On strong calls alone, the Meta set would pay off only after about 2–3 pools of this size, before counting reference labels. Please confirm or correct this and add an end-to-end comparison of this kind.
  • Other concerns. Validation across annotators and interfaces: every experiment uses the same Qwen3 family, with low Yahoo accuracies. Also, LLM routing and active label cleaning are now mentioned but not cited (e.g., Bernhardt et al., 2022; ActiveLab).
AuthorsOpenReview ↗

Response to Reviewer xkzu: Four-State Ablations and Additional Cost Analysis

Dear Reviewer xkzu,

Thank you for your careful follow-up. Your comments helped us identify an important distinction between the contribution of the signed ranking rule and that of the four-state estimator. We conducted additional analyses using the frozen experimental records, without rerunning the LLM annotators or modifying the original test results.

1. Four-state P(WC) versus SAV.

We evaluated the P(WC) ranking from the same frozen four-state model, changing only the ranking score while keeping the candidate pool, predictions, and budgets fixed. Compared with ranking by P(WC) alone, SAV achieved 73 and 121 additional net corrections at the 20% and 40% budgets, respectively. This directly supports the contribution of accounting for CW risk in the ranking rule.

We also evaluated an independently trained two-correctness baseline. It achieved net gains of +53 and +94 at the two budgets, compared with +46 and +89 for SAV. The paired confidence intervals include zero. Thus, we cannot claim that the four-state architecture is uniquely superior to independently estimated correctness probabilities. We have clarified that our main contribution concerns signed utility-aware ranking rather than the indispensability of a particular estimator architecture.

Both comparisons are post-hoc supplementary analyses on the already disclosed Test set, not new confirmatory evaluations.

2. Yahoo Cheap-error and Random utility.

We revisited the original Yahoo records and found that the cheap annotator made 3,246 errors among 4,000 examples. We also report Random's actual budget-level utility rather than significance counts alone. At the 40% matched logical-token budget, Random achieved an average net gain of +477.09. These results are now included in the supplementary analysis.

3. Meta-supervision cost.

We examined your amortization estimate. Under the call-count-based assumptions, the estimated ratio is approximately 2.64 target pools. Using the experimentally matched logical-token budget and the corresponding Random utility, the illustrative ratio is approximately 2.14 pools.

We emphasize that neither value is a measured monetary break-even point. Human reference-label costs and complete GPU billing data are unavailable, so these estimates are reported as resource-based amortization scenarios rather than demonstrated economic savings.

4. Annotator generalization and related work.

We acknowledge that the current experiments use annotators from the Qwen3 model family. Therefore, cross-model-family generalization has not yet been established. We have clarified this limitation and expanded the related-work discussion to include relevant work on LLM-based routing and active label cleaning, including Bernhardt et al. (2022) and ActiveLab.

Thank you again for identifying these issues. We believe the additional analyses make the source and scope of AdaLabel's empirical advantages clearer, while allowing us to state more precisely what the current evidence does and does not establish.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.