- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies which existing cheap annotations should be replaced by a stronger, imperfect annotator under a limited budget. It defines signed annotation value as the conditional expected improvement in label correctness, accounting for both corrections and regressions. AdaLabel estimates four joint correctness outcomes from paired reference-labeled Meta data and ranks target examples using only information available before the strong annotation.
Critical strengths
- This work introduces a well-defined, practically relevant objective with an explicit information boundary. The method distinguishes correctable errors from joint failures and explicitly penalizes harmful replacements. Learning from paired Meta outcomes while restricting deployment to cheap-side information gives a concrete implementation of prospective annotation allocation. This is more informative than treating stronger annotations as uniformly beneficial.
- Yahoo rankings are reported as frozen before target strong outcomes were accessed, with resource matching and paired uncertainty estimates. Improvements of 73–170 net corrections over Direct MLP are meaningful within that comparison. The authors also disclose the post-Gold status of DBPedia14, weak or negative comparisons, and limited downstream significance. This transparency strengthens confidence in the narrower empirical conclusions.
Critical weaknesses
- The incremental allocation benefit of the distinguishing components is insufficiently established. In the prospective Yahoo evaluation, AdaLabel improves over WC-only by 0, −4, −2, and +3 net corrections; on 20 Newsgroups, the differences are +4, +2, +8, and 0. Although the CW correction changes selections and avoids regressions, these benefits are largely offset by lost rescues. Furthermore, every four-state versus two-correctness interval includes zero, leaving the necessity of the four-state parameterization unestablished. The larger DBPedia advantage is drawn from a corrective post-Gold audit. Thus, the strongest evidence does not demonstrate a consequential allocation advantage attributable to the method’s distinctive components.
- The evaluation establishes a narrower practical advantage than the headline comparison suggests. Yahoo gains of 73–170 corrections are measured against Direct MLP, whereas gains over Direct Linear are only 13–22 and WC-only is essentially competitive; no Yahoo Cheap-error result is reported. Practical cost-effectiveness also remains unresolved because paired Meta supervision—including 2,000 reference-labeled examples—is excluded from target budgets, with no cost-inclusive or amortization analysis. The low observed annotator accuracies further motivate validation across alternative annotation interfaces. Missing discussion of modern LLM routing and active label cleaning also leaves the novelty and baseline coverage incomplete.
Questions for authors
- Beyond the annotation-construction application, what precise result distinguishes signed annotation value from conditional risk-difference allocation in budgeted triage? How should the two-correctness identity above affect the stated formulation contribution?
- Do any existing prospectively frozen comparisons show a practically meaningful, statistically supported advantage over WC-only or two-correctness estimation? Are downstream results against these alternatives available? Such evidence would materially strengthen the contribution.
- Appendix Q uses Qwen3-1.7B as the strong annotator for Meta supervision but Qwen3-8B for Gold. What motivates this change, and how should it affect interpretation of the auditability result under the stated stable-hierarchy assumption? Is there a matched binary WW detector to separate the value of four-state modeling from simply providing explicit WW supervision?
- What target-population scale is needed to amortize paired Meta annotation and reference-label costs? An accounting based on the existing experiments would help determine when the proposed allocation produces an end-to-end resource benefit.
Thank the authors for the response and the revision. The submission has improved with a narrower claim. While some of my concerns are acknowledged rather than fully resolved: