- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Chorus learns reliable supervision from heterogeneous, noisy LLM annotators without ground-truth labels in the annotation pool. It couples Bayesian label and annotator-reliability estimation with downstream model training, then uses high-confidence disagreements to provide annotator-specific feedback for subsequent batches. Experiments on five datasets spanning text classification, image classification, and closed-set visual question answering show improvements in both annotation and downstream accuracy, with additional benefits from stronger annotators.
Critical strengths
-
The paper is well written and easy to follow.
-
The method is well motivated by heterogeneous, class-dependent annotation errors. The coupling between supervision estimation, downstream training, and feedback-guided annotation is reasonable.
-
The experiments demonstrate improvements in annotation quality and downstream performance across five datasets. The ablations also support the usefulness of task-model evidence and annotator-specific feedback.
Critical weaknesses
My main concerns are gaps in the experimental evaluation:
-
Understanding how feedback improves Chorus is important. In Figure 5, since successive rounds are evaluated on different subsets, changes in feedback quality are difficult to distinguish from subset-specific difficulty or responsiveness to feedback. Evaluating feedback from successive rounds on the same diagnostic batch would help clarify whether the feedback itself improves consistently.
-
All evaluated annotators are open-source models. It remains unclear whether Chorus provides comparable benefits when advanced closed-source LLMs are used as annotation sources.
-
The additional CoT experiments are helpful, but the paper does not evaluate self-consistency [1] or repeated sampling from a single strong annotator. A comparison with multiple samples from one strong model, under a matched annotation budget, would help determine how much of the benefit comes from heterogeneous annotators rather than repeated sampling.
-
Although FreeAL [2] is already mentioned in the related work, I think it would be helpful to include an experimental comparison, as its training paradigm shares similarities with Chorus. Both methods learn from noisy labels during downstream training and use feedback from earlier rounds to improve subsequent annotation.
-
Table 2 omits the LLM-based annotation refinement baselines MAD or DiZiNER, although both are included in Table 1. Their absence leaves it unclear whether Chorus retains its advantage over these refinement methods when stronger annotators are used.
[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023
[2] FreeAL: Towards Human-Free Active Learning in the Era of Large Language Models. EMNLP 2023
Questions for authors
Please refer to the weaknesses above.
Review Author Responses
I appreciate the author's responses to clarify points raised. While these clarifications are an improvement, it does not change my rating label.