acceptodds
Under review as a conference paper at ICLR 2027

Chorus: Learning Reliable Supervision from Noisy LLM Annotators

Abstract

Large language models (LLMs) offer a scalable alternative to human annotation, but their reliability varies across models and tasks, and their errors are often class-dependent. Such heterogeneous errors make it difficult to obtain reliable supervision for downstream learning. To address this challenge, we propose Chorus, a closed-loop framework for learning reliable supervision from noisy and heterogeneous LLM annotations. It couples supervision estimation with downstream model training and uses the resulting evidence to guide subsequent LLM annotation. In Chorus, unlabeled data is processed in sequential annotation rounds. Within each round, a downstream-aware Bayesian estimation combines LLM annotations with task-model predictions, used as instance-specific label priors, to jointly infer latent labels and annotator-specific reliability. Across rounds, Chorus identifies recurring discrepancies between individual annotations and high-confidence inferred labels, and converts them into annotator-specific feedback for the next batch. In this way, Chorus operates through two coupled loops: supervision estimation and task-model training alternate within each round, while inferred supervision is converted into feedback that guides annotation across rounds. Extensive evaluations across five datasets show that Chorus improves annotation and downstream accuracy by up to 5.29% and 6.20%, respectively. With 3–8B annotators, it approaches clean-supervision performance on multiple datasets and remains competitive with methods using 72–78B annotators, while achieving further gains with stronger annotators.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
data annotation, large language models, supervised learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xzr8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Chorus learns reliable supervision from heterogeneous, noisy LLM annotators without ground-truth labels in the annotation pool. It couples Bayesian label and annotator-reliability estimation with downstream model training, then uses high-confidence disagreements to provide annotator-specific feedback for subsequent batches. Experiments on five datasets spanning text classification, image classification, and closed-set visual question answering show improvements in both annotation and downstream accuracy, with additional benefits from stronger annotators.

Critical strengths
  1. The paper is well written and easy to follow.

  2. The method is well motivated by heterogeneous, class-dependent annotation errors. The coupling between supervision estimation, downstream training, and feedback-guided annotation is reasonable.

  3. The experiments demonstrate improvements in annotation quality and downstream performance across five datasets. The ablations also support the usefulness of task-model evidence and annotator-specific feedback.

Critical weaknesses

My main concerns are gaps in the experimental evaluation:

  1. Understanding how feedback improves Chorus is important. In Figure 5, since successive rounds are evaluated on different subsets, changes in feedback quality are difficult to distinguish from subset-specific difficulty or responsiveness to feedback. Evaluating feedback from successive rounds on the same diagnostic batch would help clarify whether the feedback itself improves consistently.

  2. All evaluated annotators are open-source models. It remains unclear whether Chorus provides comparable benefits when advanced closed-source LLMs are used as annotation sources.

  3. The additional CoT experiments are helpful, but the paper does not evaluate self-consistency [1] or repeated sampling from a single strong annotator. A comparison with multiple samples from one strong model, under a matched annotation budget, would help determine how much of the benefit comes from heterogeneous annotators rather than repeated sampling.

  4. Although FreeAL [2] is already mentioned in the related work, I think it would be helpful to include an experimental comparison, as its training paradigm shares similarities with Chorus. Both methods learn from noisy labels during downstream training and use feedback from earlier rounds to improve subsequent annotation.

  5. Table 2 omits the LLM-based annotation refinement baselines MAD or DiZiNER, although both are included in Table 1. Their absence leaves it unclear whether Chorus retains its advantage over these refinement methods when stronger annotators are used.

[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023
[2] FreeAL: Towards Human-Free Active Learning in the Era of Large Language Models. EMNLP 2023

Questions for authors

Please refer to the weaknesses above.

Reviewer_P2TkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a feedback-driven framework that uses Bayesian estimation to infer label posteriors from noisy LLM annotations and iteratively refines annotations with the estimated posterior information. Experiments across five diverse datasets verify consistent gains on label estimation and downstream tasks, demonstrating favorable cross-dataset generalization.

Critical strengths
  1. The problem formulation explicitly recognizes that LLM annotations are not purely random noise and that annotators possess varying reliability. The Bayesian framework is therefore adopted to estimate the reliability of each annotation source as well as the posterior distribution of true labels, yielding a coherent logical connection between theoretical modeling and problem definition.
  2. The empirical evaluation is comprehensive. All experiments are repeated three times with mean and standard deviation reported. Baseline methods are carefully aligned for a fair comparison.
  3. Experimental results demonstrate that the proposed approach improves the quality of annotation signals, fulfilling the goal of closed-loop optimization set forth in the paper.
Critical weaknesses
  1. The figures lack readability and are weakly connected to section 3. It would be better to label them according to the subsection titles in section 3.
  2. Under sparse annotation settings, the latent labels and annotator confusion parameters are coupled. Theoretically, they cannot be uniquely distinguished: it is unclear whether the difficulty stems from hard samples or unreliable annotators.
  3. LLM annotation errors consist of random sampling noise and inherent systematic bias of the model. This paper does not design dedicated experiments to disentangle these two types of noise and evaluate the performance of Chorus under each noise type separately.
Questions for authors

The entire Chorus closed-loop pipeline adopts heuristic iteration. However, the experiments do not demonstrate whether this process is guaranteed to converge, what fixed point it converges to, whether oscillation may occur, or whether there exists a convergence upper bound.

Reviewer_h4KWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Chorus, a framework for obtaining reliable supervision from multiple noisy LLM annotators. The authors observe that different LLMs exhibit varying annotation quality across tasks and classes, and argue that this heterogeneity can be exploited rather than simply averaged away. Chorus combines two components: a downstream-aware Bayesian supervision estimation procedure that jointly infers latent labels and annotator-specific confusion matrices, and a feedback mechanism that uses high-confidence disagreements to improve subsequent LLM annotations. The framework operates over multiple annotation rounds. Within each round, inferred soft labels and downstream model predictions are updated iteratively, while feedback generated from previous rounds is used to guide annotators on new samples. Experiments across five datasets covering text classification, image classification, and visual question answering show improvements in both annotation quality and downstream accuracy compared with several aggregation and annotation refinement baselines.

Critical strengths

LLMs are increasingly used for data annotation, and their heterogeneous, class-dependent errors make reliable aggregation an important challenge. The analysis in Figure 1 provides useful evidence that different annotators have complementary strengths that cannot be captured by a single reliability score.

I also appreciate the attempt to connect supervision estimation with subsequent annotation. Instead of treating label aggregation as a separate preprocessing step, Chorus uses the inferred supervision both to train the downstream model and to guide future LLM annotations. This is a reasonable and potentially useful framework design.

The experimental results are encouraging. Chorus achieves the best reported mean annotation and downstream accuracy across all five datasets in Table 1. The authors also evaluate both smaller and larger LLM annotators, which helps demonstrate that the approach is not restricted to a particular model scale. The ablation study in Table 3 provides some evidence that task-model predictions and feedback contribute complementary improvements.

Critical weaknesses
  1. The methodological novelty is somewhat limited.

My main concern is that the two central components of Chorus are largely based on existing techniques. The Bayesian supervision estimation builds on standard probabilistic aggregation with annotator-specific confusion matrices and variational inference. Incorporating downstream model predictions into label estimation is also closely related to prior learning-from-crowds approaches that jointly estimate annotator reliability and task-model parameters. Similarly, the feedback component identifies recurring disagreements and converts them into annotator-specific instructions, which is conceptually close to existing disagreement-guided annotation refinement methods. While combining these components into a closed-loop framework is reasonable, I am not fully convinced that the integration constitutes a sufficiently substantial methodological contribution. The theoretical development does not significantly change this assessment. The two theorems in Section 2.2 mainly derive standard mean-field variational updates under the assumed probabilistic model. They establish the optimization procedure but do not appear to provide new theoretical insight into the reliability or convergence of the proposed feedback mechanism.

  1. The reliability of the feedback loop is not sufficiently established.

This is perhaps my most important technical concern. Chorus generates feedback by comparing annotator predictions against high-confidence inferred labels. However, these inferred labels are estimated using the same noisy annotations and downstream model predictions. Consequently, the framework may reinforce incorrect supervision when the inferred posterior is confidently wrong. This issue becomes particularly relevant when multiple LLM annotators make correlated errors. The probabilistic formulation assumes conditional independence of annotator outputs given the latent label and annotator reliability. In practice, LLMs may share systematic biases, which can lead to overconfident aggregated predictions even when the underlying supervision is incorrect. The authors use a confidence threshold to select feedback examples, but this does not directly address the problem of confidently incorrect predictions. Although the experiments show average improvements, I would like to see a more direct evaluation of feedback correctness, error propagation across rounds, and robustness to correlated annotator errors. Without such evidence, the reliability of the proposed closed-loop process remains somewhat uncertain.

  1. The evaluation does not adequately account for annotation and computational costs.

The paper emphasizes the practical advantage of obtaining reliable supervision from relatively small LLM annotators. However, the experimental comparisons primarily control for downstream optimization steps rather than total annotation cost. Chorus requires multiple annotators, sequential annotation rounds, iterative supervision estimation, and additional feedback instructions. These introduce computational and inference overhead that may differ substantially from those of the compared methods. In particular, the claim that Chorus with 3–8B annotators can remain competitive with approaches using 72–78B annotators is interesting, but model size alone is not sufficient to establish efficiency. A comparison in terms of total generated tokens, GPU-hours, latency, or annotation cost would make this claim much more convincing.

  1. The experiments do not fully isolate the contribution of feedback-guided refinement.

Table 3 provides a useful ablation of task-model evidence and feedback, but it only covers Irony and VQA-RAD. Given that feedback-guided annotation is one of the two main contributions, I would expect a more comprehensive analysis across the evaluated datasets. I am also interested in whether the improvement comes specifically from identifying annotator-dependent class confusions, or whether similar gains could be achieved through simpler prompt modifications. For example, providing generic class-disambiguation instructions or representative examples without the proposed posterior-based feedback selection could potentially improve annotation quality as well. The current experiments establish that feedback is helpful in the evaluated settings, but they do not fully demonstrate that the proposed feedback construction is necessary or superior to simpler alternatives.

  1. The evaluation settings leave some questions about generalization.

Although the paper evaluates five datasets, several involve relatively conventional classification tasks. The VQA-RAD evaluation is restricted to binary yes/no questions, and the framework assumes a fixed, known label space. These settings are reasonable for evaluating the proposed probabilistic model, but they do not fully reflect the more challenging annotation scenarios in which LLMs are often used, such as ambiguous semantic judgments, open-ended answers, or fine-grained annotation with correlated errors. However, I would therefore be cautious about drawing broad conclusions regarding the general reliability of Chorus across LLM annotation tasks.

Questions for authors
  1. Could the authors clarify what is methodologically new in the Bayesian supervision estimation beyond existing learning-from-crowds methods that jointly estimate annotator reliability and downstream model parameters? In particular, which aspect of the proposed formulation is not captured by prior approaches?

  2. How reliable are the high-confidence inferred labels used to generate feedback? It would be useful to report the precision of these reference labels and examine whether incorrect feedback accumulates or propagates across annotation rounds.

  3. How robust is Chorus when annotators exhibit strongly correlated errors? Since the probabilistic model assumes conditional independence, an experiment with annotators sharing systematic biases would help establish the limits of the approach.

  4. Could the authors provide a comparison under matched total annotation or computational budgets, including LLM inference cost, feedback prompt overhead, and supervision-estimation cost?

  5. How does the proposed feedback mechanism compare with simpler alternatives, such as generic class-confusion instructions or example-based prompt refinement without posterior-guided discrepancy selection?

1 comment
Reviewer_h4KWOpenReview ↗

Review Author Responses

I appreciate the author's responses to clarify points raised. While these clarifications are an improvement, it does not change my rating label.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.