acceptodds
Under review as a conference paper at ICLR 2027

Reversible Hierarchical Active Correction under Imperfect Human Feedback for MLLM-human Hybrid Annotation

Abstract

Obtaining high-quality labeled data is both expensive and time-consuming, motivating MLLM-human hybrid annotation as a scalable alternative. However, ordinary reviewers can make mistakes, while expert adjudication is too costly to apply throughout the dataset. Existing methods identify suspicious LLM-generated annotations, and selectively query human reviewers for correction. Nevertheless, they largely assume that allocating more reviewed budget to ordinary-human review should enhance annotation accuracy. In this paper, we demonstrate that this assumption fails when human reviewers are imperfect. We further identify two failure modes of selective human correction under imperfect feedback empirically, namely, human-replacement failure, where an incorrect human label overwrites a correct machine one, and candidate insufficiency, where neither the machine nor the human label is correct. To address these, we propose a Hierarchical Active Correction with Rollback (HAC-R) pipeline. Specifically, HAC-R leverages a three-stage annotation pipeline to dynamically acquire reviewed samples via a training-free label-error detection score, resolving conflicts between machine-annotation and noisy-human annotation and escalating to expert-annotation. To leverage diverse annotation source, we further design a routing-aware loss tailored to HAC-R data, with theoretical guarantees on design-based unbiasedness and routing variance. Extensive experiments on six image and text classification benchmarks demonstrate that HAC-R improves annotation accuracy and downstream clean-test accuracy by up to over cost-matched baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLM annotation, imperfect human feedback, hybrid annotation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PPvjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hierarchical Active Correction with Rollback (HAC-R), a framework that combines machine labels, imperfect human annotations, and expert adjudication under an annotation budget. A shared unreliability score guides human acquisition, selection between machine and human labels, and expert escalation. The authors introduce a downstream training objective using inverse query-probability weighting and establish unbiasedness and a routing-variance decomposition under stated assumptions. Experiments evaluate annotation accuracy and downstream performance on six image and text classification benchmarks.

Critical strengths
  1. Relevant problem formulation. The paper explicitly addresses harmful human corrections and cases where neither the machine nor the human provides the correct label. This motivates treating human feedback as an imperfect intermediate source.
  2. Connection to downstream learning. The propensity-weighted objective accounts for selective label acquisition, and the theoretical analysis clarifies how resolution and expert escalation jointly affect estimator variance.
  3. Empirical evaluation and diagnostic analyses. The experiments cover six benchmarks across two modalities, including real crowd annotations on CIFAR-100N. The component analyses are informative, and the paper acknowledges settings where rollback harms annotation quality or fails to improve downstream accuracy.
Critical weaknesses
  1. Dependence on the reliability score and limited guidance for deployment.
    The same score influences acquisition, rollback, and escalation, making its quality central to practical performance. Table 2 evaluates stage-specific behavior, and Appendix B.6 explains when rollback reduces variance. However, it remains unclear how performance changes when the underlying score becomes uninformative or systematically favors incorrect machine labels.

    This is particularly relevant when expert coverage is limited. An uninformative resolver can reject beneficial corrections when human labels are more accurate on the reviewed subset. The paper already observes harmful rollback in low-human-noise settings (Section 5.3), but provides limited guidance on validating a score or deciding when rollback should be enabled. The unbiasedness guarantee does not itself ensure low variance or improved downstream accuracy.

    Practical configuration also needs clarification: when reference labels are scarce, how should users evaluate the score and estimate the source error rates used for budget allocation, and how should the required expert-labeling cost be accounted for?

  2. The distinction from existing multi-source routing methods needs stronger support.
    The setting can be viewed as three annotation sources with different costs and error patterns; the intermediate source need not be human. Section 6 discusses related routing approaches, but the empirical comparisons primarily involve ACT/CDI with individual correction sources and the authors’ HAC variant without rollback.

    A clearer explanation of the contribution beyond existing weak/strong-labeler routing and model cascades would strengthen the paper. In particular, it would help to isolate the value of reversible candidate selection and the hierarchical training objective from improvements attributable to the scoring function or access to an additional annotation source.

  3. The demonstrated scope is limited to classification.
    All six main benchmarks use fixed-label classification. The scoring framework relies on representation compatibility and neighborhood support, with modality-specific adjustments. Its applicability to tasks requiring spatial, geometric, or open-ended judgments remains unclear—for example, defect localization in generated images or 3D assets, or assessment of open-ended captions. I view this as a limitation of the demonstrated scope rather than evidence that extension is impossible. A discussion of the required score properties and difficult application settings would improve the practical guidance.

Questions for authors
  1. How should a practitioner validate the reliability score on a new task with limited expert labels, and determine whether rollback is preferable to accepting human corrections? Can this validation cost be included in the annotation budget?
  2. Could a focused sensitivity analysis vary the reliability score’s quality while holding machine/human labels and query budgets fixed? Controlled score perturbations or alternative scoring representations could clarify the method’s dependence on this component.
  3. What distinguishes HAC-R from a comparable three-source cascade that selects between acquired labels and escalates uncertain cases? Which gains arise from the proposed routing decisions, and which arise from the downstream loss correction?
  4. What properties would a score need to support tasks beyond classification, and in which settings would constructing such a score be difficult? A discussion would be sufficient; I am not requesting an extensive new task suite.
Reviewer_KmFXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the correction of machine-generated annotations when human reviewers are imperfect and expert adjudication is costly. It identifies two failure modes: human feedback can replace a correct machine label with an incorrect one, and neither source may provide the correct label. The proposed Hierarchical Active Correction with Rollback (HAC-R) framework addresses these through three stages: acquiring human reviews for suspicious machine labels, comparing machine and human candidates using a training-free neighbourhood-based reliability score, and passing unresolved cases to experts. Empirical studies report gains in annotation and downstream accuracy over selected baselines.

Critical strengths

The paper addresses a very critical problem in human-MLLM annotation and proposes a coherent solution. It is well motivated to distinguish harmful human replacement from cases that require an additional annotation source. The proposed methods have the following two strengths:

  • HAC-R uses a shared label-conditional reliability score to acquire human reviews, compare machine and human candidates, and prioritise expert adjudication. This design distinguishes errors that can be resolved by choosing between existing labels from cases requiring additional supervision, while allowing harmful human replacements to be reversed. The mechanism operates without task-specific training or access to the machine annotator’s internal logits
  • Equation (7) combines machine, resolved, and expert losses through sequential inverse-propensity corrections. Proposition 1 establishes unbiasedness for the expert-supervised empirical risk at fixed model parameters under positive-support sequential sampling, even when intermediate annotations are incorrect. Its variance decomposition further identifies how resolution quality and expert-query probabilities jointly affect estimation precision, providing a theoretical basis for analysing the pipeline
Critical weaknesses
  • The signed loss raises a substantive optimisation concern. For an accepted human correction without expert escalation, the contribution is λ[(1−1/pi)ℓiM+ℓiH/pi]\lambda[(1-1/p_i)\ell_i^M+\ell_i^H/p_i], which can approach ∞\infty when the labels disagree and $p_i<1$. Unbiasedness therefore does not ensure a well-behaved training objective. Please clarify the safeguards used in the main experiments, since bounded-logit stabilisation is documented only for exploratory runs. The theoretical justification for rollback remains conditional. Equation (49) requires a reduction in propensity-weighted squared loss residuals, but Eq. (4) selects labels using a neighbourhood score whose connection to these residuals is unestablished. Figure 8 shows that annotation accuracy can improve while estimator variance increases under Min routing. Please assess whether the main SoftMin configuration satisfies the relevant conditions. This limits the theory’s support for the chosen resolver without invalidating the stated result.
  • The experiments do not isolate the contributions of hierarchical routing and the proposed loss. Table 2 shows that the acquisition score alone substantially changes correctable-error recall, reaching 46.9% on Cars compared with 17.7% for the criticiser used by ACT. The end-to-end comparisons therefore combine changes in scoring and routing. Table 3 also lacks a comparison between Equation (7) and ordinary cross-entropy using identical annotations and routing histories. Proposition 1 establishes fixed-parameter unbiasedness, but does not establish better downstream learning. Matched-score routing comparisons and a matched-history loss ablation are needed to substantiate the two technical contributions
Questions for authors
  • The constraint ∑iπb(si)≤b\sum_i\pi_b(s_i)\le b bounds the expected query count. Please clarify the statement that threshold sampling controls the realised count: under the stated positive-support Bernoulli rule, both sampling schemes have random query counts. Is a separate fixed-size sampling or budget-enforcement procedure used?
  • Proposition 1’s unbiasedness and routing-variance identities are valid under fixed-data sequential Bernoulli sampling. Please make the conditioning explicit and clarify that the resolver and expert propensities are determined before expert-query draws. Please also confirm that the implemented routing uses conditionally independent Bernoulli draws, since enforcing exact query counts would generally require covariance terms in Equation (8).
  • Section 3 already produces a final annotation for every example. Please clarify that the hierarchical objective in Section 4 targets the fully expert-supervised risk, whereas ordinary cross-entropy targets the retained annotations. To show the value of retaining intermediate supervision, compare final-label cross-entropy, direct machine-to-expert inverse-propensity correction, and the proposed hierarchical loss using identical annotations and routing histories. The unbiasedness identity alone does not establish a downstream advantage.
Reviewer_eg1aOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies MLLM-human hybrid annotation when the ordinary human reviewers are themselves imperfect. The authors first argue that the implicit premise of selective human correction, namely that spending more review budget produces increasingly accurate annotations, fails in practice. Using CIFAR-100N they decompose the failure into two modes: human-replacement failure, where an incorrect human label overwrites a correct machine label, and candidate insufficiency, where neither candidate is correct. They then propose HAC-R, a three-stage pipeline that acquires suspicious machine annotations for human review, resolves each conflict by comparing the two candidates under a shared label-conditional unreliability score with the option to roll back, and escalates residual cases to expert adjudication. For downstream learning they derive a hierarchical active M-estimation objective with two levels of inverse-propensity correction, prove its unbiasedness for the expert-supervised risk, and give a variance decomposition.

Critical strengths
  1. The failure-mode analysis is the most valuable part of the paper and is well evidenced. Framing human review as an imperfect, reversible intermediate source rather than as a correction oracle is a contribution, and it explains a phenomenon that practitioners observe but rarely formalize.
  2. The pipeline is a sensible systems answer to the problem, and the rollback rule is cheap. Resolve compares the machine and human candidates with the same label-conditional score and requires only their ordering within an example, so no extra model, no human confidence elicitation, and no retraining is needed.
  3. The real-crowd result on CIFAR-100N, where HAC-R improves over HAC by 1.15 points and reaches 81.53 against 76.26 for ACT, is the single most persuasive number in the paper.
Critical weaknesses
  1. My primary concern is that the problem formulation of this paper is not well-founded and lacks research significance. First, the paper's research scope is not sufficiently comprehensive: it considers data annotation but overlooks crowdsourced annotation, a widely adopted paradigm. In crowdsourced annotation, it is a well-established consensus that annotations provided by ordinary humans contain noise, and a substantial body of work has already addressed how to mitigate the impact of such noise. In particular, obtaining multiple noisy annotations for each instance through repeated annotation and then inferring the ground-truth label via label aggregation has become a mature line of research. These studies bear directly on the problem definition of this paper. Yet the paper almost entirely disregards them and still defines its research problem under a full-coverage annotation paradigm. In fact, several of the phenomena mentioned in the paper would not arise under repeated annotation, and the research space of this problem would be greatly enriched instead, such as aggregating the outputs of multiple different MLLMs, or aggregating annotations from MLLMs and ordinary humans. Moreover, the authors' solution still requires expert annotation. Even if only a subset of instances is selected for annotation, expert annotation remains infeasible for large-scale annotation tasks, in terms of both cost and efficiency.
  2. In Table 1, HAC-R is dominated by its own ablation over half of the evaluation, and there is no rule that decides when to use it. This raises a concern: does the rollback mechanism actually work at all? Or, more precisely, when is it supposed to take effect?
Questions for authors

Please see the Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.