- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies classification risk control under selective labeling and proposes NPSL, which constructs a structured admissible labeling set to compute the worst case empirical type I error and select a classification threshold. The paper provides theoretical analysis and evaluates the proposed method on synthetic and real world LendingClub data. The method addresses a meaningful problem and offers a useful approach to risk control under selective labeling. However, the relationship between the theoretical guarantees and the default algorithm, the credibility of the structural assumptions, and the conservatism of the resulting risk control remain insufficiently established.
Critical strengths
-
The paper addresses a meaningful classification risk control problem under selective labeling and introduces a structured admissible labeling set.
-
The authors exploit the structure of the labeling set to develop an exact optimization algorithm, avoiding exhaustive enumeration of all possible labelings.
-
The evaluation includes both synthetic and real-world data and investigates the effects of different selection mechanisms and structural assumptions.
Critical weaknesses
-
The theoretical analysis in Theorem 4.1 requires population class proportions to be bounded away from zero and one, whereas the default admissible set permits one class to contain only one unlabeled instance. The authors should clarify whether the default algorithm satisfies these assumptions and whether the convergence guarantees apply to the actual implementation.
-
The method relies on structural assumptions that cannot be directly verified from selectively labeled data. The theoretical justification involves unobservable class proportions and score distributions. The authors should clarify how these assumptions can be justified in practice and what risk control guarantees remain when they are violated.
-
The empirical type I error is substantially below the target in several settings, while the type II error remains considerably higher than the best achievable value. For example, with a mean separation of two, the respective values are 0.310 and 0.168. The authors should analyze the resulting loss in power and clarify the practical advantages of the proposed method over existing approaches.
-
When the structural assumptions are violated, the method may degenerate into a trivial classifier or violate the risk constraint even when the admissible set is nonempty. The paper lacks effective diagnostics for identifying potential failures in risk control. The authors should further characterize these failure modes and investigate how unreliable predictions can be detected.
-
The real data experiment evaluates only approved applicants and artificially hides labels, preventing direct validation of population risk control across all applicants. Furthermore, the evaluation uses only logistic regression based on FICO scores and debt to income ratios. The authors should discuss these limitations and provide stronger evidence for the method's applicability to practical selective labeling settings.
Questions for authors
Please refer to the Weaknesses.
3 comments
Response to Reviewer DXBY (2/2)
The empirical type I error is substantially below the target in several settings, while the type II error remains considerably higher than the best achievable value. For example, with a mean separation of two, the respective values are 0.310 and 0.168. The authors should analyze the resulting loss in power and clarify the practical advantages of the proposed method over existing approaches.
The original label in the table did not clearly identify that the "best possible" column included full knowledge of the true distributions. We have updated this to "Oracle " and updated the caption. The loss in power can be interpreted as a price to pay for the unobserved labels. Specifically, in the mean separation of two example, the cutoff given by NPSL must keep the type I error at most for the worst plausible labeling. Because the true labeling is not this worst case, the type I error is typically lower than the target. This protection against the worst case without considering every possible labeling is NPSL's practical advantage: it can provide type I error control without assuming MAR, at a smaller cost in power when compared to the all-labelings algorithm (type II risk vs at , see Figure 1). The other two baseline comparisons, IPW and fuzzy augmentation, are not robust to the MNAR case. Although it is possible that these methods could be modified to be robust to this case, perhaps using a sensitivity bound in the style of Rambachan et al. (2026), choosing this bound would require tuning in some form. This is an interesting direction for future research, and we have noted this in the conclusion.
The real data experiment evaluates only approved applicants and artificially hides labels, preventing direct validation of population risk control across all applicants. Furthermore, the evaluation uses only logistic regression based on FICO scores and debt to income ratios. The authors should discuss these limitations and provide stronger evidence for the method's applicability to practical selective labeling settings.
We have addressed this point in two ways: we have added a second real-data example on Wisconsin sentencing data, where selection is a real historical decision and the score is fit by XGBoost rather than logistic regression (Section 5.4, Appendix C.2), and clarified in Section 5.4 why we use a semi-synthetic design for LendingClub. Please see G3 in the general response for details.
References
- Rambachan, Coston, and Kennedy. "Robust design and evaluation of predictive algorithms under unobserved confounding." Review of Economics and Statistics, 2026.
Follow-up on revision
Thank you again for your review. As the discussion period closes today, we wanted to check whether our responses address your main concerns. In particular, Section 4 now states more clearly the relationship between the practical algorithm and theory, and Corollary B.8 (Appendix B.7) relaxes the class-proportion assumption. Appendix D.4 reports conditional violation rates and type II risk when the restrictions are misspecified (G2), and Section 5.4 adds a sentencing example with an XGBoost score. Please let us know if any points remain unclear.
Response to Reviewer DXBY (1/2)
We thank the reviewer for their clear review and helpful comments on the link between theory and practice in our manuscript. We have revised the manuscript to address each comment below.
As stated, the default algorithm does not satisfy these assumptions, and we have clarified this in Section 4. On the theoretical side, we have added Appendix B.7 to include Corollary B.8, which relaxes the assumptions on class proportions, including when only one unlabeled instance is in class 1 (the class 0 proportion remains bounded below in our recommended values). The corollary requires an additional assumption, equivalent to Δ1>0 when G is unbounded above. Under this assumption, Proposition 4.2 continues to hold, and Theorem 4.1 holds with a slower rate: at ε=Γ=N−1/3, the slack is of order N−1/6logN rather than N−1/3. On the practical side, we keep the recommended default the same to avoid requiring an additional parameter choice; since enlarging the admissible set can only raise the selected threshold, we believe this to be a reasonable, conservative choice.
In short, without additional assumptions, a type I error guarantee must consider every labeling of the unlabeled data (Proposition 2.1), and the restrictions use the structure of the selective labels problem to rule out implausible labelings. Please see G1 in the general response for the full explanation and the text added to Section 2.3 and the introduction.
In general, the theoretical guarantees need not hold when the assumptions are violated. We have added Appendix D.4, which studies violation rates when the assumptions are not satisfied. Please see G2 for the table and full analysis. The primary protection is the return of an empty admissible set, which suggests the data are not compatible with the assumptions. However, a nonempty admissible set does not imply that the assumptions are satisfied: in the weak reversal setting, all 11 of the 500 replications that returned a nonempty admissible set violated the type I error constraint.
Please see G2 for a more detailed analysis of the additional experiments in Appendix D.4, which studies the robustness of NPSL to structural violations. The primary diagnostic for identifying potential failures in risk control is an empty admissible set, which signals that the observed data are incompatible with the assumptions. However, as the experiments show, a nonempty admissible set does not imply the assumptions are satisfied.
In general, non-identifiability limits direct verification of the assumptions from the data alone. Beyond the aforementioned empty-set diagnostic, we have thus emphasized practical assessment (Section 2.3), which may combine domain knowledge about historical decisions with a sensitivity analysis on the data, checking the stability of the threshold and the emptiness of the admissible set.