- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The study analyzes whether improving benchmark label accuracy (using error filtering techniques) guarantees improvement in the model and operating threshold selection. The authors use a speaker verification task with different models and simulate label errors from a subset of the LibriSpeech dataset to answer the main questions. A separate budget study distinguishes fidelity to provisional labels from reference decision quality and encoder computation. A main conclusion is that a cleaner benchmark does not always support a better model and threshold selection.
Critical strengths
-
The central claim and methods appear technically sound. The paper shows that certain benchmarks can favor certain models/thresholds just because of how the label errors are distributed, and that is an illustrative contribution.
-
The evaluation targets consequential decisions and reports counterevidence. The authors report scenarios where filtering erroneous labels improves performance, making clear that their claims/findings demonstrate a possible failure of curation, rather than a general argument against filtering (although that is not very clear in the abstract and the conclusions).
Critical weaknesses
-
The conclusions come from simulated errors on small English read-speech datasets. It is unclear how this is representative of scenarios with actual labeling errors or in other settings.
-
The contribution is narrower than the broad benchmark curation scenario. Other existing studies establish correction under class-conditional corruption and highlight the importance of conditional independence (e.g, for recovering ROC curves from imperfect references). Although the application of this idea to benchmarking is somehow new and illustrative, the conclusions are not novel and the methods are applied to a very specific and narrow scenario.
Questions for authors
The first part of the related work section merely lists papers without providing a clear critique or motivation for the current work. The authors mix general papers about benchmarks with some other papers about speaker recognition/verification without making a clear distinction. Whereas this is not a question, this issue makes the motivation and impact of the work unclear.
Could the improvements in table 6 depend on how ties are resolved? IN this sense, how many improvements remain when comparing the most favorable original-placement with the least favorable shuffled errors?
Is the sampling result caused by coverage only, or by other differences between the methods?
Lastly, a general comment: The text structure and flow could be improved. Sometimes, paragraphs are just a collection of non-cohesive sentences. This text is included in Figure 1 illustrates that writing style: “Oracle cohort-specific cost correction; fixed scores, cohorts and class-error counts 20 initial seeds; each shuffled point averages 20 permutations; exploratory”.