acceptodds
Under review as a conference paper at ICLR 2027

Suspicious Labels, Useful Inputs: Cross-Fitted Soft Retention for Few-Shot Text Regression

Abstract

When training labels come from multiple raters, an unreliable label does not imply that its associated text is useless. Existing noisy-label methods often treat a high-loss example as both unreliable supervision and an input that should be removed; under limited data, these decisions need not coincide. We separate supervision reliability from input utility: a held-out predictor, decoupled from student training, estimates text–label compatibility, and this signal is converted into a continuous supervision weight so that unreliable labels have less influence while their text remains available to the student. We show theoretically that the held-out construction removes the direct self-influence of an example's label on its own residual and give conditions under which residual ranking is better than random ranking. Numerical experiments on two real multi-rater text datasets show that continuous soft retention outperforms both equal weighting and hard deletion with privileged knowledge of the corrupted examples; shuffling the correspondence between weights and examples removes the gain. The results suggest that, in limited-data noisy-label learning, label reliability should not directly determine whether an input is discarded.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
noisy-label learning, multi-rater annotation, sample reweighting, text regression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iCGVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether label reliability alone should determine whether a training text is kept. The authors propose an out-of-fold (OOF) text–label compatibility score that is computed before student training and used as a fixed example weight. The method keeps all text inputs but downweights examples whose observed labels have high OOF residuals. The paper provides a ranking bound for OOF residuals and a fixed-feature risk analysis (Proposition 2) suggesting that soft weighting can beat hard deletion under certain conditions. Experiments on two real multi-rater hate speech datasets (Measuring Hate Speech and HateXplain) show that OOF weighting achieves higher mean R^2 than equal weighting, robust baselines, and oracle hard deletion. Shuffled OOF weights remove the gain, supporting the importance of example-level correspondence. The paper also reports small-sample scaling, synthetic-noise checks, and several controls.

Critical strengths
  1. The problem framing is conceptually valuable and makes a plausible conceptual argument; however, the experiments do not independently establish text usefulness apart from labels.
  2. The method is simple, transparent, reproducible, and avoids self-reinforcing adaptive reweighting; this does not by itself establish theoretical coverage or superiority over all baselines.
  3. The shuffled and oracle controls are informative design choices; they show ESS alone does not explain the gain, but they do not resolve the unexplained OOF-vs-true-deviation-oracle gap or the limited statistical power.
Critical weaknesses
  1. Proposition 2 analyzes weighted least squares on a fixed feature map, with homoskedastic and uncorrelated clean-label noise (covariance σc2I\sigma_c^2 I) and a single common weight aa on every suspected example. The proposed method instead uses per-example out-of-fold (OOF) residual ranks and trains a four-layer Transformer end to end. The paper states this gap in Section 3.4 and again in the conclusion. Theorem 1 is a lower bound, Pr⁡(rinoisy>rjclean)≥Pr⁡(∣εi∣>2A)\Pr(r_i^{\mathrm{noisy}} > r_j^{\mathrm{clean}}) \ge \Pr(|\varepsilon_i| > 2A). Under continuous Gaussian noise, the appendix identifies the pairwise ranking probability with the residual-ranking AUC, which is the definition of AUC when ties have probability zero, and lower-bounds that probability by 2[1−Φ(A/σ)]2[1-\Phi(A/\sigma)]. The text does not set the AUC equal to this expression. The proof would be clearer if it stated that the AUC equals the pairwise probability and that this probability is at least the stated bound.

  2. Both primary benchmarks are hate-speech corpora, Measuring Hate Speech and HateXplain. As the paper notes, this is a replication across related corpora, not a test of cross-domain annotation. The Wine, Yelp, and Amazon checks use synthetic label corruption and report gains over equal weighting of +0.0490+0.0490, +0.0870+0.0870, and +0.0672+0.0672 (three seeds; p=0.0182p = 0.0182, 0.00260.0026, and 0.07370.0737). They do not establish generalization to naturally occurring multi-rater disagreement. The conclusion already acknowledges this limit.

  3. The student is a four-layer Transformer trained from scratch (width 256, 4 heads, feed-forward width 512, maximum length 128), with no pretrained text parameters. Whether the gains would remain if the student were BERT, RoBERTa, or MiniLM is untested. Frozen MiniLM appears only in Appendix Table 9, as an alternative representation for the OOF proxy (ΔR2=+0.0432\Delta R^2 = +0.0432 versus equal weighting, p=0.0543p = 0.0543), not as the student.

  4. The main comparisons use five paired seeds, and several contrasts are inconclusive. On HateXplain, OOF weighting versus Co-teaching has p=0.0746p = 0.0746; Table 5 separately reports Co-teaching versus equal weighting at p=0.0768p = 0.0768. Random hard deletion versus equal weighting has p=0.0827p = 0.0827. The Amazon result, p=0.0737p = 0.0737, comes from a three-seed synthetic check, not from the five-seed main protocol. The paper already marks some intervals as exploratory, states that the tests are unadjusted for multiple comparisons, and does not claim a definitive ranking of methods. The statistical limitation is the small number of seeds and the lack of a multiplicity correction.

  5. The trained baselines are Huber loss, Co-teaching, and Meta-Weight-Net. MentorNet and learning-to-reweight are cited in related work but are not included in the experiments, and more recent noisy-label and reweighting methods are also absent. This is a gap in the comparison set. It does not by itself invalidate the baselines that were run.

  6. On the first real benchmark, TF-IDF OOF weights have AUC 0.62320.6232 against the known corruption mask, and frozen MiniLM has AUC 0.61860.6186. Mean performance increases with the OOF blend strength, and shuffling the same weights removes the gain over equal weighting. The paper does not, however, characterize the examples that receive low weights: rater substitutions, high-disagreement cases, or hard but clean examples. That breakdown is still missing, so the link from a moderate ranking signal to the reported gains is only partly explained.

  7. OOF weighting has a higher mean R2R^2 than the true-deviation rank oracle on both benchmarks: +0.0266+0.0266 on Measuring Hate Speech (95% CI [0.0111,0.0421][0.0111, 0.0421], p=0.0088p = 0.0088) and +0.0248+0.0248 on HateXplain (95% CI [0.0053,0.0442][0.0053, 0.0442], p=0.0241p = 0.0241). That oracle ranks examples by each substituted label's deviation from the all-rater mean. It is distinct both from oracle hard deletion and from the soft mask that knows only which labels were substituted. The paper reports the gap and does not explain it. The shuffle control shows that effective sample size does not account for the gain over equal weighting (OOF ESS ≈0.750N\approx 0.750N; shuffled weights return to equal weighting). It does not explain the gap relative to the true-deviation rank oracle.

  8. Every compared student is trained with a weighted supervised loss. The paper correctly notes that this objective does not separate the contribution of the text from the contribution of the label. The experiments therefore support down-weighting suspicious labels while retaining their inputs. They do not independently establish how useful the text is apart from its label.

  9. The released implementation does not match two protocol claims that the paper’s setting and contribution rely on. Section 4.1 states that, on 30% of training comments, the multi-rater mean is replaced by one randomly selected real rater label. In run_hate_multirater.py and rater_corrupt, a score equal to that mean is removed whenever another rater score exists, so substitution is biased toward labels that already disagree with the aggregate. The resulting corruption is therefore stronger and more selected than the random rater draw described in the paper, and the reported gains are tied to this altered noise process. Separately, the abstract and Section 3.2 state that each example is scored by a text–label predictor that was not trained on it, and that its fold is excluded from the out-of-fold fit. The five-fold split excludes the example’s label from the Ridge fit, but TfidfCache fits the vocabulary and IDF on all training texts, including the text being scored. The residual is consequently not produced by a fully held-out text–label pipeline. Proposition 1 still applies to the label path because it treats the feature matrix as fixed, but the central claim that the reliability score is out-of-fold is only partly implemented.

Questions for authors
  1. The OOF-versus-equal tests are significant (Measuring Hate Speech, p=0.0098p = 0.0098; HateXplain, p=0.0006p = 0.0006). OOF versus Co-teaching (p=0.0746p = 0.0746), Co-teaching versus equal weighting (p=0.0768p = 0.0768), random hard deletion versus equal weighting (p=0.0827p = 0.0827), and the three-seed Amazon check (p=0.0737p = 0.0737) are not. Which of these contrasts does the claim depend on?

  2. The released Co-teaching code is a within-batch small-loss cross-update whose forget rate rises to 0.3. Is that the procedure in Table 5, and how does it differ from the cited Co-teaching method? The same question applies to Meta-Weight-Net.

  3. On Measuring Hate Speech, TF-IDF OOF ranking of the corruption mask has AUC 0.6232. Among the lowest-weight examples in the reported runs, what share are substituted labels, clean labels with high rater disagreement, and clean labels with low disagreement?

  4. What score does the true-deviation rank oracle assign to a clean example, and what score does it assign to a substituted example? If every clean example receives the same score, the oracle does not rank clean examples against each other, and the higher OOF R2R^2 (+0.0266+0.0266, +0.0248+0.0248) is a different comparison from beating a stronger information set.

  5. Should the contribution be judged as showing that down-weighting suspicious labels while keeping their text beats equal weighting, or as showing that the text is useful apart from its label? The reported students are trained with a weighted supervised loss.

  6. Section 4.1 replaces the multi-rater mean with one randomly selected rater score. The released code drops any score equal to that mean whenever another score exists. Which rule was used for the tables? If it was the code, do the paired gains over equal weighting remain when the substitute is drawn uniformly from all rater scores?

  7. For the reported runs, was the TF-IDF vocabulary and IDF fit inside each fold, or on all training texts? If it was fit on all training texts, how much do the OOF residuals, the weights, and the test R2R^2 change when the vectorizer is refit without the scored text?

3 comments

[W1]

In response to this concern, we revised and clarified the theoretical statements and related proofs in Section 3.4. Proposition 2 is now explicitly limited to the weighted least-squares risk analysis under a fixed feature map and the corresponding assumptions; its local risk conclusion is not a guarantee for an end-to-end Transformer. We also revised Theorem 1 and its proof to distinguish the pairwise ranking probability from its lower bound: under continuous noise and when the probability of ties is zero, the AUC equals the probability of correct pairwise ranking, while Theorem 1 provides a lower bound on that probability. See Section 3.4; the proofs are in Appendix A.1--A.3.

[W2]

We added cross-domain experiments on Yelp and Amazon review-rating tasks. The results support the effectiveness of the method across different text domains (Appendix B.5, Table 11). We also added natural multi-rater experiments on EmoBank and POPQUORN (Appendix B.8, Table 15).

[W3]

We added an experiment using a pretrained DistilBERT student. The results show that the weighting procedure can be used with a pretrained student and achieves positive results on the tasks evaluated (Appendix B.7, Table 14).

[W4]

(1) Based on the results of this study, we believe that five seeds provide statistical power, and both primary comparisons are positive across the five paired seeds.

(2) We applied Holm correction for multiple comparisons to the primary OOF-versus-Equal comparisons under the corrected protocol. The primary results remain significant after Holm correction (Table 1).

(3) Table 2 shows that OOF achieves the highest mean R2R^2 among the methods compared on both benchmarks.

(4) The three-seed synthetic Amazon experiment you mentioned has now been updated to a five-seed experiment. The results are in Appendix B.5, Table 11.

[W5]

We added two recent noisy-label baselines, NLR (2022) and ConFrag (2024). The experimental results show that OOF outperforms these two comparators in the settings evaluated (Table 6; implementation details are in Appendix B.10).

[W6]

We added an analysis of the composition of the lowest-weight examples. On Measuring Hate Speech and HateXplain, 41.5% and 38.6%, respectively, are experimentally replaced labels. The remaining low-weight examples are mainly unreplaced individual ratings that differ substantially from the multi-rater aggregate. This shows that the low weights respond not only to experimentally injected label replacements, but also identify ratings that disagree with the aggregate (Appendix B.2, Table 8).

[W7]

We recomputed the oracle using a tie-aware rule. The results show that OOF performs better than this oracle on both datasets. The oracle ranks examples by label deviation, which is not equivalent to ranking them by downstream prediction risk (Table 13).

[W8]

We agree that a supervised loss cannot independently establish the value of text apart from its label. The experiments support the conclusion that, in the settings tested, retaining examples and reducing the influence of questionable labels performs better than directly deleting low-ranked examples (Figure 3, Table 13).

[W9]

Thank you for raising this issue. We corrected the rater-sampling and TF--IDF fitting protocols. Following the paper's protocol, we sample uniformly from all observed rater scores and fit TF--IDF separately within each fold. Under the corrected protocol, the gain of OOF over Equal remains; test performance improves after fold-local fitting, and the weight ranking remains broadly stable (Section 4.1, Appendix B.1; the stability audit is in Table 16).

AuthorsOpenReview ↗

Official Comment by Authors

[Q1]

The main conclusion rests on the multiplicity-corrected OOF--Equal comparison. Co-teaching, MWN, and oracle results are supplementary comparisons of the method's applicability (Tables 1--6).

[Q2]

Yes. The Co-teaching and Meta-Weight-Net results in Table 2 of the revised manuscript correspond to these implementations. Co-teaching uses two student networks. Within each minibatch, the two networks separately select low-squared-error examples and provide their selected examples to each other for training. The forget rate increases linearly from 0 to 0.3 over the first 10 epochs. Compared with the original classification method, we retain the small-loss cross-update mechanism and replace the classification loss with squared error for regression. Meta-Weight-Net uses a loss-to-weight network with two hidden layers to generate weights from per-example squared errors. The network is updated through a virtual update on a training minibatch and a meta-loss on a clean validation minibatch, and is then used for weighted student training. These are adaptations of the original methods to regression. Implementation details are in Appendix B.10, and the comparisons are in Table 2.

[Q3]

To answer what types of examples make up the lowest-weight group, we added a composition analysis. The results show that low-weight examples include both experimentally replaced labels and original ratings that differ substantially from the multi-rater aggregate. This shows that the method not only identifies artificially injected label noise, but also captures individual ratings that disagree with the aggregate, supporting its use to estimate the relative reliability of sample labels (Appendix B.2, Table 8).

[Q4]

For each observed label, we compute the oracle score from its absolute deviation from the multi-rater mean, di=∣yi−yˉi∣d_i = |y_i - \bar y_i|, and convert −di-d_i to a mean-normalized rank weight; tied examples receive average ranks. Thus, unreplaced examples have zero deviation and are tied with one another. Replaced examples receive weights according to their deviations; if a replacement score happens to equal the multi-rater mean, it is also tied with the unreplaced examples. This oracle ranks examples by label deviation; it is not an oracle that selects weights based on downstream prediction risk. After recomputation, the paired mean ΔR2\Delta R^2 of OOF over the oracle is +0.0238 ±\pm 0.0166 on Measuring Hate Speech and +0.0324 ±\pm 0.0047 on HateXplain. See Table 2 and Appendix B.6, Table 13.

[Q5]

We position our contribution as a student-independent reliability-weighting and sample-retention procedure for subjective multi-rater text regression, whose properties and effects are examined through theoretical analysis and matched experiments. To evaluate the sample-retention strategy, we conducted a matched experiment with the OOF ranking and student model fixed, comparing continuous downweighting while retaining all inputs with deleting low-ranked examples. In the settings tested, the paired mean ΔR2\Delta R^2 of continuous downweighting over hard deletion is +0.1314 ±\pm 0.0574 on Measuring Hate Speech and +0.0398 ±\pm 0.0186 on HateXplain. The results support retaining low-weight examples and reducing their label-loss contribution over deleting examples according to the same OOF ranking. Since both strategies use supervised losses, this comparison evaluates the sample-retention strategy and does not separately identify the causal contribution of text apart from its label. See Section 3.3, Section 4.2, Figure 3, and Appendix B.6, Table 13.

[Q6]

To test whether the gain remains when rater scores are sampled uniformly according to the paper's protocol, we reran the primary experiments using that rule. The results show that OOF retains its advantage over Equal after correcting the sampling and fitting protocols (Table 1).

[Q7]

Thank you for pointing out the discrepancy between the original implementation and the protocol description. The original implementation fitted TF--IDF on all training texts before cross-validation, so the held-out text in each fold contributed to the vocabulary and IDF statistics, although its label was not used in fitting. We reran the experiments using fold-local fitting: in each fold, TF--IDF is fitted only on that fold's training texts, then applied to the held-out texts to compute OOF residuals. The Spearman correlations between the global-fit and fold-local residual rankings are 0.961 on Measuring Hate Speech and 0.955 on HateXplain; the Jaccard overlaps of the bottom-30% weight sets are 0.864 and 0.828, respectively. Test R2R^2 changes from 0.4074 ±\pm 0.0275 to 0.4166 ±\pm 0.0381 (+0.0092 ±\pm 0.0175) on Measuring Hate Speech, and from 0.3831 ±\pm 0.0253 to 0.3855 ±\pm 0.0214 (+0.0024 ±\pm 0.0071) on HateXplain. See Sections 4.1 and Appendix B.1; the audit is reported in Table 16.

Thanks for the detailed rebuttal. Two concerns remain unresolved:

  1. The mechanism attribution remains unclear: bad labels vs. hard/high-disagreement examples. Table 17 shows that down-weighting only the substituted labels while keeping all text is close to Equal and below OOF on MHS. I am aware ESS is not matched, but the mask has the higher ESS and the lower R², so ESS alone does not explain the gap.

  2. The reported numbers are inconsistent, and the paper does not map each table to its protocol. OOF on MHS takes different values across tables; Tables 7 and 9C are both labelled as the original per-seed runs but disagree; and Δ = 0.0593 ± 0.0031 in Table 2 cannot be recovered as a paired SD or SE from either.

For these reasons I keep my score at 2.

Reviewer_NDMmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors ask whether a training example with an unreliable label should be deleted or kept at a reduced loss weight in multi rater text regression. The authors fit a 5-fold TFIDF Ridge proxy, turn each example's out-of-fold residual rank into a fixed loss weight, and train a from-scratch 4-layer Transformer with these weights. The theory covers a label's effect on its own out-of-fold residual, when corrupted examples get larger residuals than clean ones and when a small weight on a suspected group beats deleting it. On Measuring Hate Speech and HateXplain, with 30% of 3,000 training comments given one real rater's label instead of the multi-rater mean, OOF weighting has the highest mean test R^2, above three baselines and two oracles. They also report that shuffled weights lose the gain, that gains are inconsistent below 512 examples, and that three synthetic-noise datasets show mean gains.

Critical strengths

I think

  1. the method is intuitive to understand and looks correct to me in construction
  2. the overall the evaluation & controls are well established. Shuffled weights with identical ESS fall back to Equal so the gain depends on which example gets which weight. Negative results at 128 and 256 examples are reported.
Critical weaknesses

My main concerns are in 2 areas:

  1. The setting seems weak and not really few-shot. The student is trained from scratch with a vocabulary from 3000 texts. Equal reaches R^2 0.36 and is negative at 128 and 256 examples (Table 3), where OOF shows no consistent gain. Line 610 mentions "stronger-student" checks I could not locate in the manuscript. If the proxy's unreported test R^2 rivals the student's, the weighting may partly act as distillation in this case.
  2. The training objective does not separately train on the retained inputs with an unlabeled or consistency objective. Large proxy residuals can reflect unreliable labels or proxy-model issues. So the improved performance does not by itself show that the method identifies corruption or extracts information from text whose label would otherwise be removed.
Questions for authors

How do hard filtering/soft weighting/target correction compare with zero substitution and with the current substitution rate? This should help distinguish benefits that arise without substitution from additional robustness to substituted labels.

1 comment

Thank you for your review. We respond to your comments below.

[W1] The main experiment examines a four-layer Transformer trained from scratch with limited training data. In Table 5, the paired mean OOF-minus-Equal differences are positive at both 128 and 256 examples; at the same time, the standard deviations reflect variability at small sample sizes. The revised manuscript also reports results with a DistilBERT student: the paired mean OOF-minus-Equal differences are +0.0049+0.0049 on Measuring Hate Speech and +0.0182+0.0182 on HateXplain. Therefore, we limit our conclusions to the students and data scales evaluated, and do not generalize the results to all few-shot settings or student models. See Sections 4.1, 4.4, and 4.6, Table 5, and Appendix B.7, Table 14.

[W2] We interpret the OOF residual as the degree of mismatch between an observed label and the prediction of a text proxy that was not trained on that example, rather than as a judgment of label correctness. The empirical conclusions of the paper focus on the training strategy: under the current replacement setting, OOF continuous weighting obtains the reported results relative to Equal weighting and hard deletion using the same OOF ranking. We do not interpret these results as determining whether each individual label is incorrect, nor do we generalize them as showing that text has value independently of its label. See Sections 3.2 and 4.2, Figure 3, Appendices B.2 and B.6, Table 13, and the Conclusion.

[Q1] Our comparisons focus on the performance of different training treatments under the current 30% rater-score replacement setting. Section 4.2 and Appendix B.6 compare continuous soft weighting with validation-selected hard deletion using the same OOF ranking; the paired mean advantages of OOF are +0.1314+0.1314 on Measuring Hate Speech and +0.0398+0.0398 on HateXplain. Appendix B.11 also compares soft weight attenuation based on the known replacement mask. These results characterize the differences between soft weighting and deletion under the current experimental setting; our conclusions are limited to this scope. See Section 4.2 and Appendices B.6 and B.11.

Reviewer_sB1mOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an out-of-fold (OOF) weighting method for few-shot text regression with noisy labels. The method uses a TF-IDF Ridge predictor to estimate label reliability through OOF residuals, which are then converted into fixed sample weights for student training. The goal is to reduce the influence of unreliable labels while retaining their corresponding text.

Critical strengths

The proposed method is simple and effective. OOF weighting is easy to implement without modifying the student architecture. The experiments include several robust learning baselines, theoretical analysis, and ablation studies that provide some evidence of its effectiveness.

Critical weaknesses
  1. Insufficient discussion of recent related work. The Introduction contains no citations, and most references are from 2021 or earlier. The paper lacks sufficient discussion of recent studies on noisy-label learning and few-shot regression, making it difficult to assess its novelty.

  2. Insufficient justification of the problem setting. The motivation for noisy labels in a few-shot setting is not convincing. When only a handful of labeled examples are available, why can their labels not be carefully verified or refined through additional annotation? The authors should explain why label noise remains a significant problem in such a limited-data setting. Moreover, the main experiments use 3,000 training examples, which does not align well with the few-shot setting.

  3. Limited methodological novelty. The method mainly combines OOF residual estimation with rank-based sample weighting, both of which are established techniques. The theoretical analysis also relies on relatively standard properties of cross-fitting and linear models. The novel contributions remain unclear.

  4. Insufficient experimental validation. The experiments rely on a small Transformer trained from scratch, without evaluating pretrained models such as BERT or RoBERTa. Furthermore, the two primary datasets are both related to hate speech, limiting the generalizability of the results.

Questions for authors
  1. The central motivation is not directly validated. The paper argues that unreliable labels can still correspond to useful text. However, the experiments do not distinguish the benefits of retaining text from those of reducing noisy-label influence. The results therefore do not directly validate the central claim.

  2. Missing comparisons with simpler alternatives. The paper lacks comparisons with in-sample residual-based weighting and hard sample selection using the same OOF ranking. Such comparisons are necessary to isolate the contributions of OOF estimation and continuous weighting.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

[W1] We sincerely apologize for not having sufficiently discussed recent work on text regression and noisy-label learning. The revised manuscript adds relevant work to the Introduction and discusses pretrained text regression, text-model fine-tuning under noisy labels, noisy-label regularization for text regression, and sample selection and reweighting in the Related Work section. We have also added experimental comparisons with NLR and ConFrag. The relevant material is in Sections 1 and 2; the NLR and ConFrag results are in Section 4.5, Table 6, and implementation details are in Appendix B.10.

[W2] (1) We do not claim that human review is infeasible. Additional review or raters can of course improve the label information, but they require annotation resources, and disagreement in subjective ratings may not be resolved into a single undisputed label through review. This paper focuses on training under a given annotation budget.

(2) We agree that 3,000 training examples should not be described as strict few-shot learning. The revised manuscript removes ``few-shot'' from the title and frames the study as multi-rater text regression under limited data. We have also added a training-size study from 128 to 3,000 examples. See Sections 1 and 4.1, and Section 4.4, Table 5.

[W3] (1) We did not previously demonstrate the method's distinctive advantage experimentally. Cross-fitting, Ridge residuals, and rank-based weighting are not algorithms newly proposed in this paper. However, the contribution of this paper is not an assembly of algorithms; it is a student-independent training procedure designed for subjective multi-rater text regression. Around this design, the paper presents three results. First, it characterizes the different effects of an individual label perturbation on in-sample and OOF residuals. Second, under explicit assumptions, it derives a lower bound on the probability that a corrupted example receives a larger residual than a clean example, and gives a local condition under which soft weighting can reduce risk relative to deletion in a fixed-feature model. Third, on real multi-rater benchmarks, it uses matched experiments to compare OOF with in-sample weighting and continuous weighting with hard deletion under the same OOF ranking, and uses a shuffled-weight control to examine the role of the example--weight correspondence.

(2) We limit the theoretical conclusions to the analyses and experiments presented in the paper. We do not claim universal algorithmic novelty for the components, nor do we claim to have established the independent value of text apart from its label.

The revised manuscript clarifies this positioning in Sections 1, 3.2, 3.4, and 4.2--4.5.

AuthorsOpenReview ↗

Official Comment by Authors

[W4] Both primary benchmarks concern hate speech, so they cannot alone establish cross-domain generalization. The revised manuscript adds a DistilBERT student experiment, in which the paired mean ΔR2\Delta R^2 of OOF over Equal improves across multiple datasets. We also conduct controlled synthetic label-replacement experiments on review-rating tasks. The natural multi-rater experiments on EmoBank and POPQUORN use real individual ratings for training and multi-rater means for evaluation, without artificially injecting label errors; both results are positive. See Section 4.6; the DistilBERT results are in Appendix B.7, Table 14; the review-rating experiments are in Appendix B.5, Table 11; and the EmoBank/POPQUORN results are in Appendix B.8, Table 15.

[Q1] Thank you for pointing out the limitation in our experiments. We added a matched experiment to directly examine this design choice: with the OOF ranking and student model fixed, we compare continuous downweighting while retaining all inputs with deleting low-ranked examples. The paired comparison shows an improvement for continuous downweighting over hard deletion. Thus, the experiment directly supports a clearly scoped conclusion: in the settings tested, retaining low-weight examples and reducing their label-loss contribution performs better than deleting examples according to the same OOF ranking. See Section 3.3, Section 4.2, Figure 3, and Appendix B.6, Table 13.

[Q2] We added two matched comparisons. The first compares OOF residual weighting with in-sample residual weighting (Figure 3). The second fixes the OOF ranking and compares continuous downweighting with deleting low-ranked examples (Appendix B.6, Table 13). The two comparisons are used to evaluate the roles of OOF residual estimation and continuous weighting, respectively.

We also note your comment that the initial manuscript was difficult to understand. We have improved the organization of the paper and added figures to clarify the presentation. Thank you for your review.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment to AC and reviewers

Dear Area Chair and Reviewers,

Thank you for your thoughtful feedback. We have submitted point-by-point responses and uploaded a revised manuscript.

The revision clarifies the label-replacement and fold-local TF–IDF protocols, reports the primary results under the revised protocol, and adds matched comparisons and supplementary evaluations with different students and datasets. We have also refined the discussion of what the OOF residuals and the empirical results do—and do not—establish.

We welcome any follow-up questions or remaining concerns during the discussion period. Thank you again for your time and consideration.

Best regards, The Authors

AuthorsOpenReview ↗

Follow-up on Rebuttal and Remaining Concerns

Dear Reviewers,

As the discussion phase will end in around 24 hours, we would sincerely appreciate it if you could let us know whether our response has adequately addressed your concerns. We're happy to clarify anything further!

Thank you so much for your contributions!

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.