Reversible Hierarchical Active Correction under Imperfect Human Feedback for MLLM-human Hybrid Annotation

Obtaining high-quality labeled data is both expensive and time-consuming, motivating MLLM-human hybrid annotation as a scalable alternative. However, ordinary reviewers can make mistakes, while expert adjudication is too costly to apply throughout the dataset. Existing methods identify suspicious LLM-generated annotations, and selectively query human reviewers for correction. Nevertheless, they largely assume that allocating more reviewed budget to ordinary-human review should enhance annotation accuracy. In this paper, we demonstrate that this assumption fails when human reviewers are imperfect. We further identify two failure modes of selective human correction under imperfect feedback empirically, namely, human-replacement failure, where an incorrect human label overwrites a correct machine one, and candidate insufficiency, where neither the machine nor the human label is correct. To address these, we propose a Hierarchical Active Correction with Rollback (HAC-R) pipeline. Specifically, HAC-R leverages a three-stage annotation pipeline to dynamically acquire reviewed samples via a training-free label-error detection score, resolving conflicts between machine-annotation and noisy-human annotation and escalating to expert-annotation. To leverage diverse annotation source, we further design a routing-aware loss tailored to HAC-R data, with theoretical guarantees on design-based unbiasedness and routing variance. Extensive experiments on six image and text classification benchmarks demonstrate that HAC-R improves annotation accuracy and downstream clean-test accuracy by up to $5\%$ over cost-matched baselines.