- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether label reliability alone should determine whether a training text is kept. The authors propose an out-of-fold (OOF) text–label compatibility score that is computed before student training and used as a fixed example weight. The method keeps all text inputs but downweights examples whose observed labels have high OOF residuals. The paper provides a ranking bound for OOF residuals and a fixed-feature risk analysis (Proposition 2) suggesting that soft weighting can beat hard deletion under certain conditions. Experiments on two real multi-rater hate speech datasets (Measuring Hate Speech and HateXplain) show that OOF weighting achieves higher mean R^2 than equal weighting, robust baselines, and oracle hard deletion. Shuffled OOF weights remove the gain, supporting the importance of example-level correspondence. The paper also reports small-sample scaling, synthetic-noise checks, and several controls.
Critical strengths
- The problem framing is conceptually valuable and makes a plausible conceptual argument; however, the experiments do not independently establish text usefulness apart from labels.
- The method is simple, transparent, reproducible, and avoids self-reinforcing adaptive reweighting; this does not by itself establish theoretical coverage or superiority over all baselines.
- The shuffled and oracle controls are informative design choices; they show ESS alone does not explain the gain, but they do not resolve the unexplained OOF-vs-true-deviation-oracle gap or the limited statistical power.
Critical weaknesses
-
Proposition 2 analyzes weighted least squares on a fixed feature map, with homoskedastic and uncorrelated clean-label noise (covariance ) and a single common weight on every suspected example. The proposed method instead uses per-example out-of-fold (OOF) residual ranks and trains a four-layer Transformer end to end. The paper states this gap in Section 3.4 and again in the conclusion. Theorem 1 is a lower bound, . Under continuous Gaussian noise, the appendix identifies the pairwise ranking probability with the residual-ranking AUC, which is the definition of AUC when ties have probability zero, and lower-bounds that probability by . The text does not set the AUC equal to this expression. The proof would be clearer if it stated that the AUC equals the pairwise probability and that this probability is at least the stated bound.
-
Both primary benchmarks are hate-speech corpora, Measuring Hate Speech and HateXplain. As the paper notes, this is a replication across related corpora, not a test of cross-domain annotation. The Wine, Yelp, and Amazon checks use synthetic label corruption and report gains over equal weighting of , , and (three seeds; , , and ). They do not establish generalization to naturally occurring multi-rater disagreement. The conclusion already acknowledges this limit.
-
The student is a four-layer Transformer trained from scratch (width 256, 4 heads, feed-forward width 512, maximum length 128), with no pretrained text parameters. Whether the gains would remain if the student were BERT, RoBERTa, or MiniLM is untested. Frozen MiniLM appears only in Appendix Table 9, as an alternative representation for the OOF proxy ( versus equal weighting, ), not as the student.
-
The main comparisons use five paired seeds, and several contrasts are inconclusive. On HateXplain, OOF weighting versus Co-teaching has ; Table 5 separately reports Co-teaching versus equal weighting at . Random hard deletion versus equal weighting has . The Amazon result, , comes from a three-seed synthetic check, not from the five-seed main protocol. The paper already marks some intervals as exploratory, states that the tests are unadjusted for multiple comparisons, and does not claim a definitive ranking of methods. The statistical limitation is the small number of seeds and the lack of a multiplicity correction.
-
The trained baselines are Huber loss, Co-teaching, and Meta-Weight-Net. MentorNet and learning-to-reweight are cited in related work but are not included in the experiments, and more recent noisy-label and reweighting methods are also absent. This is a gap in the comparison set. It does not by itself invalidate the baselines that were run.
-
On the first real benchmark, TF-IDF OOF weights have AUC against the known corruption mask, and frozen MiniLM has AUC . Mean performance increases with the OOF blend strength, and shuffling the same weights removes the gain over equal weighting. The paper does not, however, characterize the examples that receive low weights: rater substitutions, high-disagreement cases, or hard but clean examples. That breakdown is still missing, so the link from a moderate ranking signal to the reported gains is only partly explained.
-
OOF weighting has a higher mean than the true-deviation rank oracle on both benchmarks: on Measuring Hate Speech (95% CI , ) and on HateXplain (95% CI , ). That oracle ranks examples by each substituted label's deviation from the all-rater mean. It is distinct both from oracle hard deletion and from the soft mask that knows only which labels were substituted. The paper reports the gap and does not explain it. The shuffle control shows that effective sample size does not account for the gain over equal weighting (OOF ESS ; shuffled weights return to equal weighting). It does not explain the gap relative to the true-deviation rank oracle.
-
Every compared student is trained with a weighted supervised loss. The paper correctly notes that this objective does not separate the contribution of the text from the contribution of the label. The experiments therefore support down-weighting suspicious labels while retaining their inputs. They do not independently establish how useful the text is apart from its label.
-
The released implementation does not match two protocol claims that the paper’s setting and contribution rely on. Section 4.1 states that, on 30% of training comments, the multi-rater mean is replaced by one randomly selected real rater label. In run_hate_multirater.py and rater_corrupt, a score equal to that mean is removed whenever another rater score exists, so substitution is biased toward labels that already disagree with the aggregate. The resulting corruption is therefore stronger and more selected than the random rater draw described in the paper, and the reported gains are tied to this altered noise process. Separately, the abstract and Section 3.2 state that each example is scored by a text–label predictor that was not trained on it, and that its fold is excluded from the out-of-fold fit. The five-fold split excludes the example’s label from the Ridge fit, but TfidfCache fits the vocabulary and IDF on all training texts, including the text being scored. The residual is consequently not produced by a fully held-out text–label pipeline. Proposition 1 still applies to the label path because it treats the feature matrix as fixed, but the central claim that the reliability score is out-of-fold is only partly implemented.
Questions for authors
-
The OOF-versus-equal tests are significant (Measuring Hate Speech, ; HateXplain, ). OOF versus Co-teaching (), Co-teaching versus equal weighting (), random hard deletion versus equal weighting (), and the three-seed Amazon check () are not. Which of these contrasts does the claim depend on?
-
The released Co-teaching code is a within-batch small-loss cross-update whose forget rate rises to 0.3. Is that the procedure in Table 5, and how does it differ from the cited Co-teaching method? The same question applies to Meta-Weight-Net.
-
On Measuring Hate Speech, TF-IDF OOF ranking of the corruption mask has AUC 0.6232. Among the lowest-weight examples in the reported runs, what share are substituted labels, clean labels with high rater disagreement, and clean labels with low disagreement?
-
What score does the true-deviation rank oracle assign to a clean example, and what score does it assign to a substituted example? If every clean example receives the same score, the oracle does not rank clean examples against each other, and the higher OOF (, ) is a different comparison from beating a stronger information set.
-
Should the contribution be judged as showing that down-weighting suspicious labels while keeping their text beats equal weighting, or as showing that the text is useful apart from its label? The reported students are trained with a weighted supervised loss.
-
Section 4.1 replaces the multi-rater mean with one randomly selected rater score. The released code drops any score equal to that mean whenever another score exists. Which rule was used for the tables? If it was the code, do the paired gains over equal weighting remain when the substitute is drawn uniformly from all rater scores?
-
For the reported runs, was the TF-IDF vocabulary and IDF fit inside each fold, or on all training texts? If it was fit on all training texts, how much do the OOF residuals, the weights, and the test change when the vectorizer is refit without the scored text?
3 comments
Official Comment by Authors
[Q1]
The main conclusion rests on the multiplicity-corrected OOF--Equal comparison. Co-teaching, MWN, and oracle results are supplementary comparisons of the method's applicability (Tables 1--6).
[Q2]
Yes. The Co-teaching and Meta-Weight-Net results in Table 2 of the revised manuscript correspond to these implementations. Co-teaching uses two student networks. Within each minibatch, the two networks separately select low-squared-error examples and provide their selected examples to each other for training. The forget rate increases linearly from 0 to 0.3 over the first 10 epochs. Compared with the original classification method, we retain the small-loss cross-update mechanism and replace the classification loss with squared error for regression. Meta-Weight-Net uses a loss-to-weight network with two hidden layers to generate weights from per-example squared errors. The network is updated through a virtual update on a training minibatch and a meta-loss on a clean validation minibatch, and is then used for weighted student training. These are adaptations of the original methods to regression. Implementation details are in Appendix B.10, and the comparisons are in Table 2.
[Q3]
To answer what types of examples make up the lowest-weight group, we added a composition analysis. The results show that low-weight examples include both experimentally replaced labels and original ratings that differ substantially from the multi-rater aggregate. This shows that the method not only identifies artificially injected label noise, but also captures individual ratings that disagree with the aggregate, supporting its use to estimate the relative reliability of sample labels (Appendix B.2, Table 8).
[Q4]
For each observed label, we compute the oracle score from its absolute deviation from the multi-rater mean, , and convert to a mean-normalized rank weight; tied examples receive average ranks. Thus, unreplaced examples have zero deviation and are tied with one another. Replaced examples receive weights according to their deviations; if a replacement score happens to equal the multi-rater mean, it is also tied with the unreplaced examples. This oracle ranks examples by label deviation; it is not an oracle that selects weights based on downstream prediction risk. After recomputation, the paired mean of OOF over the oracle is +0.0238 0.0166 on Measuring Hate Speech and +0.0324 0.0047 on HateXplain. See Table 2 and Appendix B.6, Table 13.
[Q5]
We position our contribution as a student-independent reliability-weighting and sample-retention procedure for subjective multi-rater text regression, whose properties and effects are examined through theoretical analysis and matched experiments. To evaluate the sample-retention strategy, we conducted a matched experiment with the OOF ranking and student model fixed, comparing continuous downweighting while retaining all inputs with deleting low-ranked examples. In the settings tested, the paired mean of continuous downweighting over hard deletion is +0.1314 0.0574 on Measuring Hate Speech and +0.0398 0.0186 on HateXplain. The results support retaining low-weight examples and reducing their label-loss contribution over deleting examples according to the same OOF ranking. Since both strategies use supervised losses, this comparison evaluates the sample-retention strategy and does not separately identify the causal contribution of text apart from its label. See Section 3.3, Section 4.2, Figure 3, and Appendix B.6, Table 13.
[Q6]
To test whether the gain remains when rater scores are sampled uniformly according to the paper's protocol, we reran the primary experiments using that rule. The results show that OOF retains its advantage over Equal after correcting the sampling and fitting protocols (Table 1).
[Q7]
Thank you for pointing out the discrepancy between the original implementation and the protocol description. The original implementation fitted TF--IDF on all training texts before cross-validation, so the held-out text in each fold contributed to the vocabulary and IDF statistics, although its label was not used in fitting. We reran the experiments using fold-local fitting: in each fold, TF--IDF is fitted only on that fold's training texts, then applied to the held-out texts to compute OOF residuals. The Spearman correlations between the global-fit and fold-local residual rankings are 0.961 on Measuring Hate Speech and 0.955 on HateXplain; the Jaccard overlaps of the bottom-30% weight sets are 0.864 and 0.828, respectively. Test changes from 0.4074 0.0275 to 0.4166 0.0381 (+0.0092 0.0175) on Measuring Hate Speech, and from 0.3831 0.0253 to 0.3855 0.0214 (+0.0024 0.0071) on HateXplain. See Sections 4.1 and Appendix B.1; the audit is reported in Table 16.
Thanks for the detailed rebuttal. Two concerns remain unresolved:
-
The mechanism attribution remains unclear: bad labels vs. hard/high-disagreement examples. Table 17 shows that down-weighting only the substituted labels while keeping all text is close to Equal and below OOF on MHS. I am aware ESS is not matched, but the mask has the higher ESS and the lower R², so ESS alone does not explain the gap.
-
The reported numbers are inconsistent, and the paper does not map each table to its protocol. OOF on MHS takes different values across tables; Tables 7 and 9C are both labelled as the original per-seed runs but disagree; and Δ = 0.0593 ± 0.0031 in Table 2 cannot be recovered as a paired SD or SE from either.
For these reasons I keep my score at 2.
[W1]
In response to this concern, we revised and clarified the theoretical statements and related proofs in Section 3.4. Proposition 2 is now explicitly limited to the weighted least-squares risk analysis under a fixed feature map and the corresponding assumptions; its local risk conclusion is not a guarantee for an end-to-end Transformer. We also revised Theorem 1 and its proof to distinguish the pairwise ranking probability from its lower bound: under continuous noise and when the probability of ties is zero, the AUC equals the probability of correct pairwise ranking, while Theorem 1 provides a lower bound on that probability. See Section 3.4; the proofs are in Appendix A.1--A.3.
[W2]
We added cross-domain experiments on Yelp and Amazon review-rating tasks. The results support the effectiveness of the method across different text domains (Appendix B.5, Table 11). We also added natural multi-rater experiments on EmoBank and POPQUORN (Appendix B.8, Table 15).
[W3]
We added an experiment using a pretrained DistilBERT student. The results show that the weighting procedure can be used with a pretrained student and achieves positive results on the tasks evaluated (Appendix B.7, Table 14).
[W4]
(1) Based on the results of this study, we believe that five seeds provide statistical power, and both primary comparisons are positive across the five paired seeds.
(2) We applied Holm correction for multiple comparisons to the primary OOF-versus-Equal comparisons under the corrected protocol. The primary results remain significant after Holm correction (Table 1).
(3) Table 2 shows that OOF achieves the highest mean R2 among the methods compared on both benchmarks.
(4) The three-seed synthetic Amazon experiment you mentioned has now been updated to a five-seed experiment. The results are in Appendix B.5, Table 11.
[W5]
We added two recent noisy-label baselines, NLR (2022) and ConFrag (2024). The experimental results show that OOF outperforms these two comparators in the settings evaluated (Table 6; implementation details are in Appendix B.10).
[W6]
We added an analysis of the composition of the lowest-weight examples. On Measuring Hate Speech and HateXplain, 41.5% and 38.6%, respectively, are experimentally replaced labels. The remaining low-weight examples are mainly unreplaced individual ratings that differ substantially from the multi-rater aggregate. This shows that the low weights respond not only to experimentally injected label replacements, but also identify ratings that disagree with the aggregate (Appendix B.2, Table 8).
[W7]
We recomputed the oracle using a tie-aware rule. The results show that OOF performs better than this oracle on both datasets. The oracle ranks examples by label deviation, which is not equivalent to ranking them by downstream prediction risk (Table 13).
[W8]
We agree that a supervised loss cannot independently establish the value of text apart from its label. The experiments support the conclusion that, in the settings tested, retaining examples and reducing the influence of questionable labels performs better than directly deleting low-ranked examples (Figure 3, Table 13).
[W9]
Thank you for raising this issue. We corrected the rater-sampling and TF--IDF fitting protocols. Following the paper's protocol, we sample uniformly from all observed rater scores and fit TF--IDF separately within each fold. Under the corrected protocol, the gain of OOF over Equal remains; test performance improves after fold-local fitting, and the weight ranking remains broadly stable (Section 4.1, Appendix B.1; the stability audit is in Table 16).