- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether retraining a selective predictor changes which of the samples are sent to review due to high uncertainty. The authors define allocation reproducibility (AR) as the overlap of top-k review sets and describe a sharp rank bound related to spearman correlation. They evaluate seven settings (six visual and one textual), each with ten retrainings. At a 10% budget, they find that roughly 38–50% of review slots turn over in the vision settings despite very low accuracy standard deviations, versus about 11% on Civil Comments. The turnover reaches persistent errors and cases whose predicted class never changes.
Critical strengths
-
The problem is well-defined and the authors choose the right experimental protocol to test it. I don't necessarily agree with the choice of models and datasets (six visions against only a textual one), but the authors span their experiments through multiple seeds, which is appreciated. The authors also released code in their supplementary material.
-
I commend the authors for the formal rigor and general clarity of the paper. From the introduction, the problem and problem setting are well presented.
Critical weaknesses
I list two critical issues that are core to my opinion. The first one is the applicability, or impact, of the study described in the paper. As per the author words, the field has long studied the effect on these classifiers with no interest in what samples exactly are sent for the review. However, the authors do not argue at all in the paper on why doing so is important. What has the field missed, so far? Why is studying the turnover of samples sent to review important? I fail to see the impact of this study, and its general framing in the literature, and thus the contribution appears to be marginal. Moreover, the paper does not present itself as a reusable benchmark, but more like an empirical study. This doubles down on the importance of making the reader understand what the impact will be.
My second concern is more technical. The authors use Corollary 1 as a "Spearman-certified overlap". To my understanding, this reverses Theorem 1, and says that given a strict-ranking Spearmak correlation , we can find an max amount of review slots that have changed. The authors use the example of N=10,000 and k=1,000 with despite zero top-10% overlap. However, to my understanding, this example applies to the paper’s own results at 10% budget. The Spearman values in Table 1 are all below the threshold. So at the main budget, Corollary 1 guarantees no overlap at all in any setting, while the observed AR is 0.50–0.89. Since Corollary 1 is presented as a way to certify overlap, I would ask the authors to report what Corollary 1 actually guarantees on their data, next to the observed AR. If it is not guaranteeing anything, it should be reframed. In general, the paper spends a lot of space formalizing things, but it's not clear how many are actually needed.
Also, to compute the strict Spearman correlation between two runs, you need both rankings. If you have both rankings, can't you just count how many review slots match? What is the use of a bound?
Questions for authors
I refer the authors to the main two weaknesses that I raised.
We clarify the practical significance of allocation reproducibility, report the empirical limitations of the Spearman bound, and distinguish its theoretical role from direct overlap measurement.
We thank the reviewer for recognizing the formal rigor and experimental protocol.
C1. Why does allocation reproducibility matter?
Aggregate performance does not reveal which cases receive limited review resources. Under fixed capacity, reviewing one case excludes another. Retraining can change these assignments even when accuracy and review capacity remain similar.
For example, after a content-moderation model is replaced, a persistently misclassified toxic comment may no longer receive human review despite similar aggregate accuracy. At the 10% budget, 37.6%–50.1% of review slots are reassigned across six visual settings. Among cases misclassified in at least eight of ten retrainings, 58.75%–83.97% change review status at least once. On Civil Comments, 27.26% of 2,487 persistently misclassified toxic comments change review status.
AR measures assignment consistency, while case-level analyses identify affected cases. We do not assume that every reassignment is harmful or that stability should always be maximized.
The released code provides a reusable evaluation protocol across models, uncertainty scores, and review budgets. We will clarify its practical motivation and scope.
C2. What does Corollary 1 actually guarantee?
The reviewer is correct that Corollary 1 provides no positive overlap guarantee for any of the 315 model pairs at the primary 10% budget.
Table 1 reports average-tie Spearman correlations, whereas the corollary requires strict rankings. We recomputed the bounds from the original predictions using deterministic tie-breaking across all 1,575 model-pair–budget comparisons. All 35 setting–budget AR estimates were reproduced, with no bound violations.
At the 10% budget:
The lower bounds are from Corollary 1; positive pairs have a strictly positive lower bound. A zero lower bound does not imply zero overlap; it means that Corollary 1 certifies no positive overlap for that comparison.
Across all budgets, 287/1,575 comparisons (18.22%) yield positive guarantees, exclusively at 20% and 30%. For Civil Comments, the mean lower bounds are 0.3399 and 0.5599, respectively, with 45/45 positive pairs at both budgets.
Mathematical sharpness does not imply empirical informativeness. We will report these results and present Corollary 1 as a worst-case bound, explicitly acknowledging its vacuity at the primary budget.
C3. Why use a bound when overlap can be computed directly?
Direct top-k overlap is exact and preferable when both rankings are available. Corollary 1 offers no computational advantage.
Theorem 1 instead establishes what global ranking agreement alone guarantees about allocation. Its sharp construction shows that, with 10,000 cases and a 10% budget, Spearman correlation near 0.988 can coexist with completely disjoint selections. This explains why global ranking agreement cannot generally substitute for direct allocation evaluation.
Proposition 3 serves a different purpose: it localizes possible membership changes through realized score and cutoff movements. Its post-hoc bounds are nontrivial in 1,519/1,575 comparisons (96.44%), with mean clipped slack 0.0583.
We will distinguish these roles, report the empirical limitations of Corollary 1, and streamline the theoretical presentation.