AdaLabel: Learning Signed Annotation Value for Budgeted Data Annotation

Stronger annotators can improve labels on average without being correct on every example. Replacing a cheap label can rescue an error or regress a correct annotation. We therefore define signed annotation value (SAV) as the expected change in annotation correctness caused by an upgrade, conditional only on information available before the strong call. AdaLabel learns the CC/WC/WW/CW outcome distribution from paired Meta supervision and ranks upgrades by $V(x)=P(WC\mid s_{\mathrm{cheap}}(x))-P(CW\mid s_{\mathrm{cheap}}(x))$. In an independently and prospectively frozen Yahoo Answers evaluation, AdaLabel improves over the prespecified Direct MLP signed-value comparator by 73–170 net corrections under matched logical-token budgets, while remaining competitive with strong heuristic alternatives. An exploratory frozen-record replay recovers WC-only under constant predicted regression risk and produces dataset-dependent ranking divergence as heterogeneity is restored, without guaranteeing utility gains. Exploratory frozen-record attribution supports instance-level value learning beyond class-only priors, and independent Amazon results show that retaining four outcomes improves joint-failure auditability. Results vary across budgets and datasets, so they do not establish universal dominance.