acceptodds
Under review as a conference paper at ICLR 2027

The Curation Paradox: Cleaner Labels can be harder to correct

Abstract

Cleaner labels and broader coverage are appealing goals for benchmark curation, but neither guarantees better model selection. We study speaker verification, where evaluation chooses both a model and an acceptance threshold. Agreement filtering can tie surviving label errors to model scores, violating the independence required by class-confusion correction. We express the remaining cost distortion as a covariance residual and test its consequences using frozen public speaker models with simulated label errors. An oracle intervention redistributes errors while preserving the cohort and error counts, isolating the role of error placement. Consistent identity merges and splits also produce cases where filtering improves labels but worsens the selected decision, alongside cases where filtering helps. For developers with only provisional labels, a separate budget study compares uniform, agreement, and score-stratified sampling. Broader observable coverage does not reliably improve decisions; closer agreement with the full provisional benchmark can preserve its errors. Moreover, selecting fewer pairs may save little encoder computation because pairs share recordings. These controlled results motivate evaluating curation by decision quality, information requirements, and processing cost, rather than treating label agreement or subset size as quality certificates.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, dataset curation, label noise, evaluation bias, speaker verification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HewLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The study analyzes whether improving benchmark label accuracy (using error filtering techniques) guarantees improvement in the model and operating threshold selection. The authors use a speaker verification task with different models and simulate label errors from a subset of the LibriSpeech dataset to answer the main questions. A separate budget study distinguishes fidelity to provisional labels from reference decision quality and encoder computation. A main conclusion is that a cleaner benchmark does not always support a better model and threshold selection.

Critical strengths
  • The central claim and methods appear technically sound. The paper shows that certain benchmarks can favor certain models/thresholds just because of how the label errors are distributed, and that is an illustrative contribution.

  • The evaluation targets consequential decisions and reports counterevidence. The authors report scenarios where filtering erroneous labels improves performance, making clear that their claims/findings demonstrate a possible failure of curation, rather than a general argument against filtering (although that is not very clear in the abstract and the conclusions).

Critical weaknesses
  • The conclusions come from simulated errors on small English read-speech datasets. It is unclear how this is representative of scenarios with actual labeling errors or in other settings.

  • The contribution is narrower than the broad benchmark curation scenario. Other existing studies establish correction under class-conditional corruption and highlight the importance of conditional independence (e.g, for recovering ROC curves from imperfect references). Although the application of this idea to benchmarking is somehow new and illustrative, the conclusions are not novel and the methods are applied to a very specific and narrow scenario.

Questions for authors

The first part of the related work section merely lists papers without providing a clear critique or motivation for the current work. The authors mix general papers about benchmarks with some other papers about speaker recognition/verification without making a clear distinction. Whereas this is not a question, this issue makes the motivation and impact of the work unclear.

Could the improvements in table 6 depend on how ties are resolved? IN this sense, how many improvements remain when comparing the most favorable original-placement with the least favorable shuffled errors?

Is the sampling result caused by coverage only, or by other differences between the methods?

Lastly, a general comment: The text structure and flow could be improved. Sometimes, paragraphs are just a collection of non-cohesive sentences. This text is included in Figure 1 illustrates that writing style: “Oracle cohort-specific cost correction; fixed scores, cohorts and class-error counts 20 initial seeds; each shuffled point averages 20 permutations; exploratory”.

Reviewer_SrJ9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how benchmark curation affects model selection. They show that agreement filtering can make the benchmark cleaner, but may still lead to a worse choice of model. Mathematically, the authors show that, after agreement filtering, the noise-corrected evaluation cost still retains a covariance residual between the decisions of the curator and the evaluated model, which directly explains why exact noise correction can fail. The authors also validate this with experiments and show that redistributing the remaining errors within each class, while keeping the number of errors fixed, reduces decision regret.

Critical strengths
  1. The paper analyzes a practical scenario in benchmark curation, where agreement filtering removes many label errors but does not necessarily lead to better model selection.

  2. The authors provide both mathematical analysis and controlled experiments to support this claim.

Critical weaknesses
  1. The presentation and organization of the paper need improvement. The main text provides only limited technical explanations and well below the ICLR page limit. The paper reads like an unpolished AI-written draft rather than a well-polished research paper.

  2. The novelty appears limited. The paper does not introduce a new method, and its contribution is primarily an analysis of a narrowly defined setting.

  3. It is currently not entirely clear which claims the experimental results are intended to support.

  4. The assumptions and theorems are not stated or organized clearly enough to assess whether the theoretical analysis supports the paper’s central conclusions.

Questions for authors
  1. Could the authors provide a more detailed theoretical analysis to directly support the paper's core claims? It would greatly improve clarity if all assumptions and theorems were explicitly stated and structured.

  2. Could the authors clarify exactly which claims their experimental results are intended to support? Currently, the experimental section is very difficult to follow.

Reviewer_7EP5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses label cleaning in benchmarks for speaker verification. Usually, to get rid of noisy labels, curator models can be applied where labels are filtered out where the model disagrees with the original label. Another way is to do class-conditioned noise correction. However, the authors postulate that if the curator model agrees with “incorrect” labels in the original dataset when filtering, so errors are not removed, the class-conditioned noise correction will still not be effective at all for the errors. The authors test whether this is actually the case.

They use five different models for the experiments and use MinDFC (tradeoff between incorrectly identifying two different speakers as the same and not identifying a single speaker as the same) as the evaluation metric.

Critical strengths
  • I like the idea of testing whether the combination of these two benchmark cleaning methods actually lead to better labels.
  • The practical considerations are really nice and showcase the importance and strength of this study. This would be very useful for researchers who want to curate their benchmark with cleaner labels. It is essential to then know whether these methods actually do what they “promise” they will or whether we are actually still stuck with noisy labels.
Critical weaknesses
  • I personally found the paper difficult to follow. While it could be due to my lack of expertise in speaker verification, I had to reread the paper multiple times to actually grasp it properly. I think the lack of a storyline might add to this, I felt the paper was jumping between the different issues but there is a lack of clear connection between the different threads it is tying together. What would definitely help is some kind of a figure that could give an idea of the experimental setup. The reader can be taken more by the hand instead of assuming that they are aware of all the different components, making it more accessible to readers who are not an expertise in speaker verification (or label cleaning in general).
Questions for authors

N/A.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.