acceptodds
Under review as a conference paper at ICLR 2027

NPSL: Tractable and Robust Classification under Selective Labels

Abstract

Many binary classification problems involve learning from data with partially missing labels. In the selective label setting, a prior decision determines which labels are observed. For example, banks observe repayment outcomes only for applicants who were granted a loan. Error control is then difficult because the observations with labels may not be representative of all observations. We study classification in this setting using the Neyman–Pearson (NP) framework, which seeks to control the prioritized type I error while minimizing the type II error. We propose NPSL (Neyman–Pearson under Selective Labels), using structural assumptions about the unlabeled population to construct a set of plausible labelings. Given an independently trained scoring function, NPSL selects a threshold that ensures the empirical type I error is at most a user-specified target across all plausible labelings simultaneously. Although the number of possible labelings grows exponentially, we show that this can be done in polynomial time. Under additional conditions, NPSL converges to the optimal robust threshold — the smallest threshold that still satisfies the type I error constraint across all plausible distributions. Synthetic experiments empirically show that NPSL provides reliable type I error control while achieving lower type II error than benchmark methods that satisfy the NP constraint. A case study using loan application data illustrates risk control of NPSL with artificially hidden outcomes.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Neyman-Pearson classification, asymmetric classification, selective labels, reject inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DXBYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies classification risk control under selective labeling and proposes NPSL, which constructs a structured admissible labeling set to compute the worst case empirical type I error and select a classification threshold. The paper provides theoretical analysis and evaluates the proposed method on synthetic and real world LendingClub data. The method addresses a meaningful problem and offers a useful approach to risk control under selective labeling. However, the relationship between the theoretical guarantees and the default algorithm, the credibility of the structural assumptions, and the conservatism of the resulting risk control remain insufficiently established.

Critical strengths
  1. The paper addresses a meaningful classification risk control problem under selective labeling and introduces a structured admissible labeling set.

  2. The authors exploit the structure of the labeling set to develop an exact optimization algorithm, avoiding exhaustive enumeration of all possible labelings.

  3. The evaluation includes both synthetic and real-world data and investigates the effects of different selection mechanisms and structural assumptions.

Critical weaknesses
  1. The theoretical analysis in Theorem 4.1 requires population class proportions to be bounded away from zero and one, whereas the default admissible set permits one class to contain only one unlabeled instance. The authors should clarify whether the default algorithm satisfies these assumptions and whether the convergence guarantees apply to the actual implementation.

  2. The method relies on structural assumptions that cannot be directly verified from selectively labeled data. The theoretical justification involves unobservable class proportions and score distributions. The authors should clarify how these assumptions can be justified in practice and what risk control guarantees remain when they are violated.

  3. The empirical type I error is substantially below the target in several settings, while the type II error remains considerably higher than the best achievable value. For example, with a mean separation of two, the respective values are 0.310 and 0.168. The authors should analyze the resulting loss in power and clarify the practical advantages of the proposed method over existing approaches.

  4. When the structural assumptions are violated, the method may degenerate into a trivial classifier or violate the risk constraint even when the admissible set is nonempty. The paper lacks effective diagnostics for identifying potential failures in risk control. The authors should further characterize these failure modes and investigate how unreliable predictions can be detected.

  5. The real data experiment evaluates only approved applicants and artificially hides labels, preventing direct validation of population risk control across all applicants. Furthermore, the evaluation uses only logistic regression based on FICO scores and debt to income ratios. The authors should discuss these limitations and provide stronger evidence for the method's applicability to practical selective labeling settings.

Questions for authors

Please refer to the Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer DXBY (1/2)

We thank the reviewer for their clear review and helpful comments on the link between theory and practice in our manuscript. We have revised the manuscript to address each comment below.


The authors should clarify whether the default algorithm satisfies these assumptions and whether the convergence guarantees apply to the actual implementation.

As stated, the default algorithm does not satisfy these assumptions, and we have clarified this in Section 4. On the theoretical side, we have added Appendix B.7 to include Corollary B.8, which relaxes the assumptions on class proportions, including when only one unlabeled instance is in class 1 (the class 0 proportion remains bounded below in our recommended values). The corollary requires an additional assumption, equivalent to Δ1>0\Delta_1>0 when GG is unbounded above. Under this assumption, Proposition 4.2 continues to hold, and Theorem 4.1 holds with a slower rate: at ε=Γ=N−1/3\varepsilon=\Gamma=N^{-1/3}, the slack is of order N−1/6log⁡NN^{-1/6}\log N rather than N−1/3N^{-1/3}. On the practical side, we keep the recommended default the same to avoid requiring an additional parameter choice; since enlarging the admissible set can only raise the selected threshold, we believe this to be a reasonable, conservative choice.


The theoretical justification involves unobservable class proportions and score distributions. How can these assumptions be justified in practice, and what risk control guarantees remain when they are violated?

In short, without additional assumptions, a type I error guarantee must consider every labeling of the unlabeled data (Proposition 2.1), and the restrictions use the structure of the selective labels problem to rule out implausible labelings. Please see G1 in the general response for the full explanation and the text added to Section 2.3 and the introduction.

In general, the theoretical guarantees need not hold when the assumptions are violated. We have added Appendix D.4, which studies violation rates when the assumptions are not satisfied. Please see G2 for the table and full analysis. The primary protection is the return of an empty admissible set, which suggests the data are not compatible with the assumptions. However, a nonempty admissible set does not imply that the assumptions are satisfied: in the weak reversal setting, all 1111 of the 500500 replications that returned a nonempty admissible set violated the type I error constraint.


When the structural assumptions are violated, the method may degenerate into a trivial classifier or violate the risk constraint even when the admissible set is nonempty. The paper lacks effective diagnostics for identifying potential failures in risk control. The authors should further characterize these failure modes and investigate how unreliable predictions can be detected.

Please see G2 for a more detailed analysis of the additional experiments in Appendix D.4, which studies the robustness of NPSL to structural violations. The primary diagnostic for identifying potential failures in risk control is an empty admissible set, which signals that the observed data are incompatible with the assumptions. However, as the experiments show, a nonempty admissible set does not imply the assumptions are satisfied.

In general, non-identifiability limits direct verification of the assumptions from the data alone. Beyond the aforementioned empty-set diagnostic, we have thus emphasized practical assessment (Section 2.3), which may combine domain knowledge about historical decisions with a sensitivity analysis on the data, checking the stability of the threshold and the emptiness of the admissible set.


AuthorsOpenReview ↗

Response to Reviewer DXBY (2/2)

The empirical type I error is substantially below the target in several settings, while the type II error remains considerably higher than the best achievable value. For example, with a mean separation of two, the respective values are 0.310 and 0.168. The authors should analyze the resulting loss in power and clarify the practical advantages of the proposed method over existing approaches.

The original label in the table did not clearly identify that the "best possible" column included full knowledge of the true distributions. We have updated this to "Oracle R1R_1" and updated the caption. The loss in power can be interpreted as a price to pay for the unobserved labels. Specifically, in the mean separation of two example, the cutoff given by NPSL must keep the type I error at most 0.150.15 for the worst plausible labeling. Because the true labeling is not this worst case, the type I error is typically lower than the target. This protection against the worst case without considering every possible labeling is NPSL's practical advantage: it can provide type I error control without assuming MAR, at a smaller cost in power when compared to the all-labelings algorithm (type II risk 0.1730.173 vs 0.5440.544 at n=1000n=1000, see Figure 1). The other two baseline comparisons, IPW and fuzzy augmentation, are not robust to the MNAR case. Although it is possible that these methods could be modified to be robust to this case, perhaps using a sensitivity bound in the style of Rambachan et al. (2026), choosing this bound would require tuning in some form. This is an interesting direction for future research, and we have noted this in the conclusion.


The real data experiment evaluates only approved applicants and artificially hides labels, preventing direct validation of population risk control across all applicants. Furthermore, the evaluation uses only logistic regression based on FICO scores and debt to income ratios. The authors should discuss these limitations and provide stronger evidence for the method's applicability to practical selective labeling settings.

We have addressed this point in two ways: we have added a second real-data example on Wisconsin sentencing data, where selection is a real historical decision and the score is fit by XGBoost rather than logistic regression (Section 5.4, Appendix C.2), and clarified in Section 5.4 why we use a semi-synthetic design for LendingClub. Please see G3 in the general response for details.

References

  • Rambachan, Coston, and Kennedy. "Robust design and evaluation of predictive algorithms under unobserved confounding." Review of Economics and Statistics, 2026.
AuthorsOpenReview ↗

Follow-up on revision

Thank you again for your review. As the discussion period closes today, we wanted to check whether our responses address your main concerns. In particular, Section 4 now states more clearly the relationship between the practical algorithm and theory, and Corollary B.8 (Appendix B.7) relaxes the class-proportion assumption. Appendix D.4 reports conditional violation rates and type II risk when the restrictions are misspecified (G2), and Section 5.4 adds a sentencing example with an XGBoost score. Please let us know if any points remain unclear.

Reviewer_mMH2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NPSL calibrates an independently trained binary-classification score when historical decisions selectively reveal labels. It restricts plausible labelings through class proportions and score distributions, then selects the smallest threshold satisfying a worst-case empirical type I error constraint. The paper provides an exact polynomial-time algorithm, establishes convergence to a robust population threshold under additional assumptions, and evaluates the method using simulations and artificially masked LendingClub outcomes.

Critical strengths
  1. The framework directly incorporates uncertainty about missing labels into threshold calibration through explicit restrictions on class proportions and score distributions. Its compatibility with independently trained scoring functions makes it applicable across classifiers, while the restrictions provide a transparent way to express the domain assumptions supporting error control.
  2. The paper exploits the monotone structure of cumulative class counts to compute the worst-case empirical risk exactly in polynomial time. The accompanying consistency result identifies the robust population threshold approached under stated assumptions, providing a substantive computational and statistical foundation for the method.
Critical weaknesses
  1. The overall theory does not directly prove that the recommended calibration default values are reasonable. It assumes a fixed and correctly set completion set and additionally assumes sampling relaxation, while the actual implementation limitations are data-dependent and their conservativeness has not been established. Although the paper acknowledges this difference, it is unclear how to select the limitations and calibration margins to achieve the user-specified finite sample violation probability. This limits the practical interpretation of the core error control objective of this method. 2.The experimental comparison fails to sufficiently distinguish the robustness to selective labels from the differences in uncertainty calibration. IPW and fuzzy enhanced baselines use plug-in empirical risk to select thresholds without performing sampling uncertainty correction, and even under MAR, they exhibit a high violation rate. It is necessary to compare under the common violation probability objective or along the matching risk-futility curve to determine whether the advantage of NPSL still exists when facing alternative methods with appropriate calibration.
Questions for authors
  1. Can the author provide a calibration procedure that controls the probability of exceeding the overall first-type error target at the user-specified level under explicit assumptions? What is the relationship between its parameter selection and sampling correction and the recommended data-dependent default values? 2.How does NPSL compare with uncertainty-adjusted IPW or fuzzy enhanced under the common violation probability objective, especially under these methods' applicable MAR? 3.Can the author extend the error setting analysis to cases where the allowable set is still non-empty and simultaneously report the conditional first-type violation rate and second-type risk? This will help clarify when this method still remains useful for risk control when the restrictions are incorrect.
3 comments
AuthorsOpenReview ↗

Response to Reviewer mMH2 (1/2)

We thank the reviewer for their thorough review and insightful comments on baseline comparisons. We have revised the manuscript to address each point below.


The population theory does not directly justify the recommended calibration defaults. It assumes a fixed, correctly specified completion set and additional sampling slack, whereas the implemented restrictions are data-dependent and their conservativeness is not established. [...] it remains unclear how to choose the restrictions and calibration margins to obtain a user-specified finite-sample violation probability. This limits the practical interpretation of the method's central error-control objective.

The defaults target a user-specified type I risk rather than a user-specified violation probability, and sampling uncertainty enters through δ0\delta_0, δ1\delta_1, and γ\gamma. Because the decomposition of the distributions of the unlabeled data into class 00 and class 11 is not possible from the data alone, the restrictions cannot be calibrated from the data to a target violation probability; any such guarantee must be conditional on assumed values of Δ0\Delta_0 and Δ1\Delta_1 (see Proposition 2.1, Section 2.3, and G1). The defaults are empirically evaluated in Section 5.3 and Appendices D.3-D.4, including in settings that violate the restrictions. There, we find that under mild violations of the assumptions, NPSL can remain robust and still provide some protection. When the violations are more severe, NPSL can provide some additional protection by returning an empty admissible set. However, we emphasize that a nonempty admissible set does not guarantee the assumptions are satisfied (see G2).

We agree that Theorem 4.1 and Proposition 4.2 do not cover the recommended defaults, and have clarified this immediately after Proposition 4.2. NPSL is constructed to control the worst-case empirical type I risk directly, and the theoretical results require the slack variables to extend this control to the population level. For a user-specified type I error target, practitioners may choose these recommended defaults, for which we find empirical support for type I error control. Users with more specific domain knowledge may change the parameters; larger values of δ0\delta_0, δ1\delta_1, and γ\gamma give more conservative thresholds, and Appendix D.2 studies the resulting trade-off. In our fourth response, we have addressed how a user-specified finite-sample violation probability could be targeted instead.


The experimental comparison does not adequately separate robustness to selective labels from differences in uncertainty calibration. The IPW and fuzzy-augmentation baselines select thresholds using plug-in empirical risks without sampling-uncertainty corrections, and they exhibit elevated violation rates even under MAR.

We have replaced the plug-in IPW and fuzzy augmentation baselines with bootstrapped versions in the two experiments where they were used (Section 5.1 and Appendix D.1). Each selects the 9090th percentile of the bootstrapped thresholds, corresponding to the target 10%10\% violation rate. The relevant violation rates and average type II risk are provided in the table below.

MethodMAR, n=1000n=1000MAR, n=10000n=10000MNAR, n=1000n=1000MNAR, n=10000n=10000
NPSL0%0\% / 0.2320.2320%0\% / 0.1820.1820%0\% / 0.1730.1730%0\% / 0.1390.139
IPW (bootstrap)0%0\% / 0.2420.2421.2%1.2\% / 0.1040.1041.4%1.4\% / 0.2110.21156.0%56.0\% / 0.0720.072
Fuzzy augmentation (bootstrap)4.0%4.0\% / 0.1350.1359.4%9.4\% / 0.0810.08137.8%37.8\% / 0.1020.102100%100\% / 0.0590.059

Violation rate / mean type II risk, α=0.15\alpha=0.15.

Under MAR, the two methods are able to obtain more reliable type I error control. When the sample size is large (n=10000)(n=10000), this makes more explicit the price paid by NPSL's robustness, as it is generally more conservative than IPW and fuzzy augmentation. We also note that when the sample size is modest (n=1000)(n=1000), the violation rates for all three methods are low, but the range of outcomes is larger for IPW and fuzzy augmentation (seen in Figure 2, Appendix D.1).

Moving to the MNAR setting now more effectively isolates the effect of selection. When n=10000n=10000, violation rates are high for IPW and fuzzy augmentation while remaining low for NPSL. At n=1000n=1000, violation rates are elevated for fuzzy augmentation. IPW has a low violation rate in this setting, but has a higher type II risk than NPSL, which has no violations in any of these settings. These additional conclusions are stated in Section 5.1.


AuthorsOpenReview ↗

Response to Reviewer mMH2 (2/2)

Comparisons at a common violation-probability target, or along matched risk-power curves, are needed to establish whether NPSL's advantage persists against appropriately calibrated alternatives. How does NPSL compare with uncertainty-adjusted IPW or fuzzy augmentation at a common violation-probability target, particularly under MAR where those methods are applicable?

We have included a common violation-probability comparison in Appendix D.6, by running each method at different user-specified α\alpha, and comparing results at matched violation rates. We note that this cannot be done in practice, as the choice depends on the true violation rate, which is unknowable. The results for the MAR setting are summarized below. At n=1000n=1000, NPSL has lower type II risk than IPW and fuzzy augmentation at every matched violation rate in both the MAR and MNAR settings. At n=10000n=10000, it stays below IPW, and is close to fuzzy augmentation, though it is slightly higher at some matched violation rates (in the MAR setting, 0.0990.099 for NPSL vs 0.0870.087 for fuzzy at a violation rate of 1%1\%, the largest difference).

nnMethod1%1\%5%5\%10%10\%
10001000NPSL0.1460.1460.1210.1210.1100.110
IPW (bootstrap)0.2000.2000.1620.1620.1400.140
Fuzzy augmentation (bootstrap)0.1490.1490.1310.1310.1220.122
1000010000NPSL0.0990.0990.0830.0830.0830.083
IPW (bootstrap)0.1060.1060.0960.0960.0910.091
Fuzzy augmentation (bootstrap)0.0870.0870.0830.0830.0810.081

Average type II risk at matched violation rates of 1%1\%, 5%5\%, and 10%10\% under MAR (Appendix D.6).

It is not immediately apparent how to adjust fuzzy-augmentation or IPW to account for their failure in the MNAR case. One technique would be to use a sensitivity bound, similar to Rambachan et al. (2026), but this would require a choice of the sensitivity parameter, which itself must be based on domain knowledge. NPSL avoids making this choice by imposing our structural assumptions, but we agree that this is an interesting research direction to pursue.


Can the authors provide a calibration procedure that controls the probability of exceeding the population type I error target at a user-specified level under explicit assumptions? How would its parameter choices and sampling corrections relate to the recommended data-dependent defaults?

Theorem 4.1, applied at level α−ε−1/N\alpha-\varepsilon-1/N, bounds the probability under explicit assumptions, but its constants are unspecified, so it cannot be set to a user-specified β\beta. It is possible to modify NPSL slightly to allow for a user-specified β\beta, though the corresponding theoretical guarantee requires a different proof technique. We have briefly outlined the technique below, also found in Appendix E. Some parameter choices would still be required to provide these guarantees, and the recommended data-dependent defaults would remain a reasonable place to start. Obtaining a full guarantee would require applying the NP-umbrella correction and choosing the values to ensure the true labeling is generally contained in the admissible set.

First, observe that if the true labeling lies in the admissible set, then combining an NP-umbrella style sampling correction with a distributionally robust approach of taking the worst-case threshold would automatically provide the desired theoretical guarantee. One could then bound the probability that the true labeling lies outside the admissible set conditional on the assumptions being satisfied, using, for example, DKW inequalities. Additional work would be required to show that these bounds are not vacuous, but we believe this is an interesting direction for future research.


Can the authors extend the misspecification analysis to settings in which the admissible set remains nonempty, reporting both conditional type I violation rates and type II risk? This would clarify when the method retains useful risk control under incorrect restrictions.

We have added Appendix D.4 to report both conditional type I violation rates and type II risk. In short, NPSL remains robust under mild violations, with conditional violation rates under 1%1\%, but in the weak reversal setting all 1111 of the 500500 replications that returned a nonempty admissible set violated the type I error constraint. Please see G2 in the general response for the table and full analysis.

References

  • Rambachan, Coston, and Kennedy. "Robust design and evaluation of predictive algorithms under unobserved confounding." Review of Economics and Statistics, 2026.
AuthorsOpenReview ↗

Follow-up on revision

Thank you again for your detailed review. As the discussion period closes today, we wanted to check whether our responses address your main concerns. In particular, IPW and fuzzy augmentation are now bootstrapped (Section 5.1, Appendix D.1), and Appendix D.6 compares all three methods at matched violation rates. We have also clarified after Proposition 4.2 what the theory does and does not cover for the recommended defaults, and Appendix E outlines a route to a user-specified violation probability. Please let us know if any points remain unclear.

Reviewer_4jrVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes NPSL for Neyman–Pearson classification under selective labels. By imposing structural constraints on plausible labelings, NPSL enables efficient worst-case Type I error calibration without relying on the MAR assumption. The authors establish theoretical guarantees and evaluate the method on synthetic and loan application data.

Critical strengths

The paper addresses an important problem and provides an efficient algorithm for exact worst-case empirical risk computation. Theoretical analysis and experiments further support the effectiveness of the proposed approach under the considered assumptions.

Critical weaknesses
  1. Strong assumptions and limited generalizability. The proposed structural assumptions require specific relationships between the scoring function, historical selection decisions, and class-conditional score distributions. However, the scoring function is trained on selectively labeled data, and these relationships are difficult to verify for unlabeled instances. Moreover, the assumptions are primarily motivated by loan approval scenarios, raising concerns about generalizability. The experiments largely construct selection mechanisms related to the scoring function, while the LendingClub evaluation uses artificially hidden labels. Although assumption violations are partially investigated, the results do not sufficiently establish reliability under naturally occurring selection mechanisms.

  2. Insufficient clarity of presentation. The paper is difficult to follow in several places. In particular, the motivation for the four structural constraints and their specific formulations is not sufficiently developed. Additionally, the relationship between empirical risk control, population-level theoretical guarantees, and practical parameter choices could be explained more clearly.

Questions for authors

Please check the weaknesses section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4jrV (1/2)

We thank the reviewer for their careful reading of our paper and helpful comments on the clarity of presentation. We have revised the manuscript to address each point below, and would welcome any further suggestions on parts that remain unclear.


The proposed structural assumptions require specific relationships between the scoring function, historical selection decisions, and class-conditional score distributions. [...] these relationships are difficult to verify for unlabeled instances

We recognize that the relationship between the structural assumptions, scoring function, historical selection decision, and score distributions is complex and not sufficiently explained. In short, without additional assumptions, a type I error guarantee must consider every labeling of the unlabeled data (Proposition 2.1), and the restrictions use the structure of the selective labels problem to rule out implausible labelings. Please see G1 in the general response for the complete explanation and additions to Section 2.3 and the introduction.


[...] the assumptions are primarily motivated by loan approval scenarios, raising concerns about generalizability.

We agree that we originally relied too heavily on the loan approval scenario for exposition. The introduction now expands on sentencing decisions to provide an explicit second example of this structure, and Section 2.3 notes other settings where these restrictions may be reasonable. Additionally, we have provided a more intuitive summary of our structural assumptions, and outlined why they may be applicable to other selective label settings, again using sentencing decisions as a concrete example (full text shown in G1).


[...] the scoring function is trained on selectively labeled data [...]. The experiments largely construct selection mechanisms related to the scoring function [...]

Although we allow the scoring function to be trained on selectively labeled data, the theoretical guarantees are conditioned on any fixed scoring function, and in particular hold when the scoring function is trained in this way. However, it is true that this can make the score less informative on the unlabeled data, and our assumptions on the admissible set are designed to ensure that the score remains at least somewhat informative. Our numerical experiments in Sections 5.1 and 5.2 directly test this point, as the scoring function is trained only on a subset of the labeled data, and we have clarified this in Section 5.1. Our results show that despite being trained on selectively labeled data, NPSL is able to effectively control the type I error when run on the full population.

We have added in Appendix D.3 additional scenarios where the selection mechanism depends on the scoring function less explicitly, in the score-correlation setting. We find that even when the selection mechanism does not depend on the scoring function, the violation rates remain low. This supports the idea that NPSL is compatible with a range of scoring functions and selection mechanisms, regardless of their correlation.


[...] the LendingClub evaluation uses artificially hidden labels. [...] the results do not sufficiently establish reliability under naturally occurring selection mechanisms.

We have addressed this point in two ways: we have added a second real-data example on Wisconsin sentencing data, where selection is a real historical decision (Section 5.4, Appendix C.2), and clarified in Section 5.4 why we use a semi-synthetic design for LendingClub. Please see G3 in the general response for more details.


The motivation for the four structural constraints and their specific formulations is not sufficiently developed.

We have revised the manuscript to better motivate the restrictions (see G1 for the full text). Each of the four structural constraints is motivated individually in Section 2.3, with the broad intuition provided at the beginning of that section: the historical decision and scoring function should point in the same direction, and the score should remain informative on the unlabeled set. We hope that the additional context of the intuition makes the individual motivations clearer. We also state in Section 2.3 (immediately before Equation 2.4) that the specific forms of our restrictions are chosen to make the computation tractable. In particular, they allow admissibility to be checked by binary search rather than checking all 2m2^m labelings (Section 3).


AuthorsOpenReview ↗

Response to Reviewer 4jrV (2/2)

[...] the relationship between empirical risk control, population-level theoretical guarantees, and practical parameter choices could be explained more clearly.

We have first updated the notation, using capital Greek letters for the population and theoretical values and lowercase Greek letters for our recommended defaults. We have also added the text below to further clarify this relationship in Section 4, immediately after Proposition 4.2.

Added to Section 4. We pause here to clearly distinguish the guarantees provided by the theoretical results from the practical implementation. By construction, NPSL ensures U^0(t^)≤α\widehat U_0(\widehat t)\le\alpha, controlling the worst-case empirical type I risk. The theoretical results motivate the recommended parameter settings given in Section 2.3, and the population bounds are related to the recommended parameters through δ0=Δ0+ε\delta_0=\Delta_0+\varepsilon, δ1=Δ1+ε\delta_1=\Delta_1+\varepsilon, and γ=Γ\gamma=\Gamma. The recommended values are constructed to account for these slack variables (Appendix D.2). However, because Δ0\Delta_0 and Δ1\Delta_1 are unknown, we cannot check whether G\mathcal G contains the true distribution, and the theoretical results do not guarantee population type I error control under the recommended values. We empirically verify these choices in Section 5.3 by fixing the default values and evaluating their type I errors across settings, including those that violate our assumptions. There, we find that the recommended values are generally robust, providing empirical evidence for population type I error control. Appendix E outlines a possible route to an extension of NPSL that provides finite-sample guarantees for a user-specified target violation rate.

AuthorsOpenReview ↗

Follow-up on revision

Thank you again for your review. As the discussion period closes today, we wanted to check whether our responses address your main concerns. In particular, Section 2.3 now further motivates the four restrictions, adding other settings where they may apply (G1), and Section 5.4 adds a second real-data example on Wisconsin sentencing data (G3), in which the selection is a real historical decision. Please let us know if any points remain unclear.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Revisions

Summary of Revisions

We thank all three reviewers for their comments, and have uploaded a revised manuscript and supplementary material. Relevant changes to the manuscript are shown in blue. Some of the major updates include:

  • Clarified motivations for structural assumptions. Proposition 2.1 shows that without restrictions, a type I guarantee must cover every labeling of the unlabeled data. Section 2.3 now gives an example of an implausible labeling, an intuitive summary of the four restrictions, and other settings where they may apply. The introduction previews this point.
  • Theory and the recommended defaults. Capital Greek letters now denote population quantities and lowercase letters the recommended defaults. In Section 4, immediately after Proposition 4.2, we explicitly state what the theoretical results do and do not cover.
  • Bootstrapped baselines. IPW and fuzzy augmentation are replaced by bootstrapped versions in Section 5.1 and Appendix D.1. The new Appendix D.6 compares all methods at matched violation rates.
  • Misspecified restrictions. The added Appendix D.4 reports overall and conditional violation rates and type II risk when the structural assumptions are violated. Section 2.3 discusses assessing the assumptions in practice.
  • Second real-data example. Section 5.4 and Appendix C.2 add Wisconsin sentencing data, where selection is a real historical decision and the score is fit by XGBoost.
AuthorsOpenReview ↗

General Response (1/2)

General Response

Here, we have gathered some responses that are applicable to multiple reviewers' comments for convenience.

G1. Motivation for the structural assumptions

Due to the missing labels, the population distribution of the unlabeled data cannot be decomposed into the class 0 and class 1 components from the data alone. Thus, any guarantee on the type I error without additional assumptions must consider all possible labelings, which can be overly conservative (see the added Proposition 2.1). However, many possible labelings may be implausible, which motivates using the structure of the selective labels problem to restrict this set.

We have added text, shown below, to Section 2.3 that explicitly provides a concrete example of an implausible labeling, why it may be implausible, and how it may be applicable to a broader set of domains. We have also added a shorter preview of this point in the introduction.

Added to Section 2.3. To build intuition, we outline a brief example of an implausible labeling. Consider a labeling that assigns the lowest unlabeled scores to class 1 and the highest to class 0. This labeling produces a large type I error and, in view of Proposition 2.1, should be considered when designing a classification rule with type I error guarantees. However, this labeling requires the score to rank the unlabeled observations in exactly the incorrect order. If the score remains informative on the unlabeled observations, it may be possible to use this assumption to restrict the set of plausible labelings.

Roughly speaking, the restrictions require the historical decision to favor class 1 and higher scores. Additionally, the score should remain informative among the unlabeled observations. While these restrictions were originally motivated by the loan approval setting, they may be reasonable in other selective label settings, such as in sentencing and hiring decisions. For example, when predicting recidivism (the chance a defendant who is released goes on to commit another crime), outcomes are typically observed only when a defendant is not jailed, and judges may tend to jail defendants they consider more likely to reoffend.


G2. Violations of the assumptions

In general, the theoretical guarantees need not hold when the assumptions are violated. We have added Appendix D.4, which studies violation rates when the assumptions are not satisfied. The primary protection is the return of an empty admissible set, which suggests the data are not compatible with the assumptions. However, a nonempty admissible set does not imply that the assumptions are satisfied. Table 4 in Appendix D.4 thus reports overall violation rates and violation rates conditional on returning a nonempty set. The table and following analysis are reproduced below.

SettingNonempty (of 500)Overall viol.Cond. viol.Cond. mean R1R_1
Class 0 selection: score coefficient
22 (baseline)5005000.0%0.0\%0.00%0.00\%0.4390.439
00 (flat)5005000.2%0.2\%0.20%0.20\%0.3810.381
−1-1 (weak reversal)11112.2%2.2\%100.00%100.00\%0.1820.182
−2-2 (strong reversal)000.0%0.0\%––
Class proportion misspecification: q0−q1q_0-q_1
004944940.6%0.6\%0.61%0.61\%0.4520.452
0.10.14434430.4%0.4\%0.45%0.45\%0.4350.435
0.20.21921920.2%0.2\%0.52%0.52\%0.4260.426
0.30.316160.0%0.0\%0.00%0.00\%0.4400.440

Behavior under restriction misspecification (Appendix D.4).

When the class proportion assumption (the first assumption) is violated, NPSL is fairly robust, with overall and conditional violation rates remaining very low (under 1%1\%) in both mild and severe misspecifications. The other three assumptions are more closely related as they all involve relationships between the unknown distributions G0G_0 and G1G_1, which are related through G=pG0+(1−p)G1G=pG_0+(1-p)G_1. We thus violate these assumptions by changing the class 0 selection mechanism, making it increasingly likely that lower scores will be labeled. The flat setting violates the class 11 displacement and ordering restrictions. There, NPSL is robust, with a violation rate of 0.2%0.2\% and never returns an empty set. The weak and strong reversal settings additionally violate the class 00 displacement restriction. In the strong reversal setting, NPSL always returns an empty set, and NPSL remains safe by returning the trivial classifier and signaling that the assumptions may be violated. In the weak reversal setting, although the overall violation rate is 2.2%2.2\%, the conditional violation rate is 100%100\%. That is, out of the 11/50011/500 replications that returned a nontrivial classifier, each one resulted in a violation of the type I error constraint. We thus emphasize that practitioners should ensure the assumptions are reasonable in their setting before using NPSL.


AuthorsOpenReview ↗

General Response (2/2)

G3. Real-data evaluation

We have added a second real-data example on Wisconsin sentencing data (Section 5.4, Appendix C.2), where selection is a real historical decision, as a case is considered labeled when the judge imposes no jail sentence. For sentences of up to 700700 days, recidivism is still recorded, so the type I risk may be evaluated on the unlabeled cases. Additionally, this example uses an XGBoost score rather than logistic regression.

We have also clarified the motivation for the semi-synthetic design in the LendingClub example of Section 5.4: this is a known difficulty in studying the selective label problem, as under most naturally occurring selection mechanisms, the outcomes of unlabeled observations are never recorded, which prevents validation of type I errors. We thus implement a commonly used semi-synthetic design, artificially hiding labels whose outcomes are known (Ehrhardt et al., 2021; Chen et al., 2025).

References

  • Ehrhardt, Biernacki, Vandewalle, Heinrich, and Beben. "Reject inference methods in credit scoring." Journal of Applied Statistics, 48(13-15):2734-2754, 2021.
  • Chen, Li, and Mao. "Learning with selectively labeled data from multiple decision-makers." ICML, PMLR 267:8480-8519, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.