- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Hierarchical Active Correction with Rollback (HAC-R), a framework that combines machine labels, imperfect human annotations, and expert adjudication under an annotation budget. A shared unreliability score guides human acquisition, selection between machine and human labels, and expert escalation. The authors introduce a downstream training objective using inverse query-probability weighting and establish unbiasedness and a routing-variance decomposition under stated assumptions. Experiments evaluate annotation accuracy and downstream performance on six image and text classification benchmarks.
Critical strengths
- Relevant problem formulation. The paper explicitly addresses harmful human corrections and cases where neither the machine nor the human provides the correct label. This motivates treating human feedback as an imperfect intermediate source.
- Connection to downstream learning. The propensity-weighted objective accounts for selective label acquisition, and the theoretical analysis clarifies how resolution and expert escalation jointly affect estimator variance.
- Empirical evaluation and diagnostic analyses. The experiments cover six benchmarks across two modalities, including real crowd annotations on CIFAR-100N. The component analyses are informative, and the paper acknowledges settings where rollback harms annotation quality or fails to improve downstream accuracy.
Critical weaknesses
-
Dependence on the reliability score and limited guidance for deployment.
The same score influences acquisition, rollback, and escalation, making its quality central to practical performance. Table 2 evaluates stage-specific behavior, and Appendix B.6 explains when rollback reduces variance. However, it remains unclear how performance changes when the underlying score becomes uninformative or systematically favors incorrect machine labels.This is particularly relevant when expert coverage is limited. An uninformative resolver can reject beneficial corrections when human labels are more accurate on the reviewed subset. The paper already observes harmful rollback in low-human-noise settings (Section 5.3), but provides limited guidance on validating a score or deciding when rollback should be enabled. The unbiasedness guarantee does not itself ensure low variance or improved downstream accuracy.
Practical configuration also needs clarification: when reference labels are scarce, how should users evaluate the score and estimate the source error rates used for budget allocation, and how should the required expert-labeling cost be accounted for?
-
The distinction from existing multi-source routing methods needs stronger support.
The setting can be viewed as three annotation sources with different costs and error patterns; the intermediate source need not be human. Section 6 discusses related routing approaches, but the empirical comparisons primarily involve ACT/CDI with individual correction sources and the authors’ HAC variant without rollback.A clearer explanation of the contribution beyond existing weak/strong-labeler routing and model cascades would strengthen the paper. In particular, it would help to isolate the value of reversible candidate selection and the hierarchical training objective from improvements attributable to the scoring function or access to an additional annotation source.
-
The demonstrated scope is limited to classification.
All six main benchmarks use fixed-label classification. The scoring framework relies on representation compatibility and neighborhood support, with modality-specific adjustments. Its applicability to tasks requiring spatial, geometric, or open-ended judgments remains unclear—for example, defect localization in generated images or 3D assets, or assessment of open-ended captions. I view this as a limitation of the demonstrated scope rather than evidence that extension is impossible. A discussion of the required score properties and difficult application settings would improve the practical guidance.
Questions for authors
- How should a practitioner validate the reliability score on a new task with limited expert labels, and determine whether rollback is preferable to accepting human corrections? Can this validation cost be included in the annotation budget?
- Could a focused sensitivity analysis vary the reliability score’s quality while holding machine/human labels and query budgets fixed? Controlled score perturbations or alternative scoring representations could clarify the method’s dependence on this component.
- What distinguishes HAC-R from a comparable three-source cascade that selects between acquired labels and escalates uncertain cases? Which gains arise from the proposed routing decisions, and which arise from the downstream loss correction?
- What properties would a score need to support tasks beyond classification, and in which settings would constructing such a score be difficult? A discussion would be sufficient; I am not requesting an extensive new task suite.