acceptodds
Under review as a conference paper at ICLR 2027

AUC Maximization from Biased Positive-unlabeled Data with Confidence

Abstract

Maximizing the area under the receiver operating characteristic curve (AUC) is a standard approach to imbalanced binary classification. Although positive and negative data are required for maximizing the AUC, negative data are often difficult to collect in some real-world applications due to privacy concerns or the need for specialized expertise to annotate them. Thus, AUC maximization from positive and unlabeled (PU) data has been attracting attention. Existing methods assume that labeled positive data are unbiased samples from the true positive distribution. However, this ideal assumption is often violated in practice. In this paper, we propose a method to maximize the AUC from biased PU data. To address the bias, our key idea is to exploit \it confidence, i.e., the probability that an instance is positive, associated with the small number of labeled positive data. %and often available in practice. We derive an estimator of the AUC risk using biased PU data with confidence, enabling AUC maximization under such bias. We further show that the rewritten AUC risk induces a Bayes-optimal AUC ranking even when the available confidence is any strictly increasing transformation of the true posterior probability. We experimentally show the effectiveness of our method on eight real-world datasets.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
AUC maximization, Biased PU learning, Confidence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rKEyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper concerns AUC maximisation problem for modified biased PU model (single-sample scenario) when additional sample of covariates for which posterior probability of positive class is given. The Authors use standard method of replacing theoretical AUC with AUC based on logistic function which approximates indicator function appearing in the definition of AUC for score under study and derive explicit formula for AUC in terms of observable quantities: probability distribution of positive class, of general population, posteriors of being in positive class and being labeled , and unobservable prior which does not affect maximisation of AUC. The manuscript contains the result that minimizer of the AUC does not change when posterior is transformed by increasing transformation (Proposition 4.1) and some numerical evidence that the method works (in terms of yielding larger AUC) when compared with scores obtained by other approaches, mostly based on SCAR assumption.

Critical strengths

Deriviation of explicit formula for for modified AUC in this setting is novel and may be useful.

The paper is clearly written and easy to follow.

Critical weaknesses

The main weakness is that assumption of availability of posterior values for additional samples is hardly realistic and makes the problem much easier: estimation of the posterior for positive class is the main hurdle for the PU problem.
The approach and thus experimental study is based on simple-sample scenario, whereas all considered competitors (save PG) were derived for case-control scenario, and, importantly, most of them are designed for unbiased setting. The Authors do not consider standard approaches of estimating posterior in PU setting such as e.g. SAREM, LBE, VAE-PU and JERM.

Questions for authors
  1. The statement of the model in (8) is confusing as labeled and unlabeled sequences are split and do not pertain to one iid sequence. In particular, formula for a^lpha=NP/(NP+N)\hat alpha=N^P/(N^P +N) yields reasonable estimator of α\alpha only in the setting when (x_i,o_i) iid sample is observable: here, one can freely take N:=2N for observable data, without changing any parameters of the model, but influencing asymptotic behaviour of α\alpha and minimizer of AUC.
  2. Line 289: It seems that π\pi can be estimated in this context by simple averaging of p^(y=1∣xi)\hat p(y=1|x_i).
  3. It would be informative to know how the obtained scores translate into classifier larger accuracies of corresponding classifiers. Was it attempted ?
5 comments

We appreciate the reviewer's thoughtful comments and valuable suggestions.

The main weakness is that assumption of availability of posterior values for additional samples is hardly realistic and makes the problem much easier: estimation of the posterior for positive class is the main hurdle for the PU problem.

Thank you for raising this important concern. We agree that estimating posterior probabilities is a fundamental challenge in standard PU learning and that positive-confidence provides additional supervision. However, our work considers a different learning setting, where confidence is available only for biased labeled positive instances, rather than for arbitrary unlabeled or test instances.

As discussed in the Introduction, such confidence can arise from existing annotation or operational processes. Furthermore, our method does not require exact posterior probabilities: Proposition 4.1 establishes that an arbitrary strictly increasing transformation of the true posterior probability is sufficient to recover the Bayes-optimal AUC ranking. We also demonstrate the applicability of our method using human-derived confidence in Table 2.

Our goal is not to solve the conventional PU learning problem, but to introduce and investigate a new weakly supervised learning setting that exploits available positive-confidence to address selection bias. We believe that studying how such additional information can be utilized for AUC maximization under SAR constitutes a meaningful research problem, distinct from conventional PU learning.

The approach and thus experimental study is based on simple-sample scenario, whereas all considered competitors (save PG) were derived for case-control scenario, and, importantly, most of them are designed for unbiased setting. The Authors do not consider standard approaches of estimating posterior in PU setting such as e.g. SAREM, LBE, VAE-PU and JERM.

Thank you for this comment. We would like to clarify the following points.

First, although some of the comparison methods were originally developed for the case-control scenario, we explicitly adapted nnPU, PUAUC, PUSB, and NPU to the one-sample scenario by exactly rewriting expectations over the marginal distribution using the labeled-positive and unlabeled distributions, as described in Appendix E.3. Therefore, these methods were not directly applied using their original case-control objectives. We will emphasize this point in the revised paper.

Second, our comparisons are not limited to methods assuming SCAR. PUSB (Kato et al., 2019) and PG (Gerych et al., 2022) are specifically designed to address selection bias. Moreover, we already compared our method with LBE (Gong et al., 2021), a representative SAR-based PU learning method applicable to the one-sample scenario, in Appendix F.6 (Table 7). Our method outperformed LBE on all six datasets. We will make this comparison more prominent in the revised paper.

  1. The statement of the model in (8) is confusing as labeled and unlabeled sequences are split and do not pertain to one iid sequence. In particular, formula for α^=Np/(N+Np)\hat{\alpha}=N^p/(N+N^p) yields reasonable estimator of α\alpha only in the setting when (x_i,o_i) iid sample is observable: here, one can freely take N:=2N for observable data, without changing any parameters of the model, but influencing asymptotic behaviour of α\alpha and minimizer of AUC.

Thank you for pointing out this ambiguity. We clarify that our problem setting follows the one-sample scenario, where V=Np+NV=N^p+N instances are initially sampled i.i.d. from the joint distribution p(x,o)p(\mathbf{x},o) and then divided into labeled positive and unlabeled data according to oo.

Thus, NpN^p and NN are determined by the original sampling process and cannot be chosen arbitrarily. In particular, α^=Np/(Np+N)\hat{\alpha}=N^p/(N^p+N) is the empirical proportion of instances with o=1o=1 and is a consistent estimator of α=p(o=1)\alpha=p(o=1) by the law of large numbers.

Doubling NN alone, as suggested in the comment, would change the sampling protocol and would no longer correspond to an i.i.d. sample from the same joint distribution p(x,o)p(\mathbf{x},o).

Eq. (8) describes the conditional distributions of the two groups after splitting the original sample. We agree that this distinction was not sufficiently clear and will revise Section 4.1 to explicitly describe the original i.i.d. sampling process and the resulting sample counts.

  1. Line 289: It seems that π\pi can be estimated in this context by simple averaging of p^(y=1∣xi)\hat p(y=1|x_i).

Thank you for this comment. Indeed, the class-prior π\pi can be estimated by averaging the posterior probabilities p(y=1∣x)p(y=1|\mathbf{x}) over samples drawn from the marginal distribution p(x)p(\mathbf{x}). However, in our problem setting, confidence values are available only for biased labeled positive instances, as defined in Eq. (8). Therefore, simply averaging the available confidence values does not generally yield π\pi, since the labeled positive distribution pl(x)p^l(\mathbf{x}) differs from the marginal distribution p(x)p(\mathbf{x}).

Furthermore, when the observed confidence is a strictly increasing transformation of the true posterior probability, as considered in Proposition 4.1, even averaging the confidence values over the marginal distribution does not generally yield π\pi.

  1. It would be informative to know how the obtained scores translate into classifier larger accuracies of corresponding classifiers. Was it attempted ?

Thank you for this suggestion. We have not evaluated classification accuracy, as our work specifically focuses on AUC maximization under class imbalance. While AUC evaluates ranking performance independently of the decision threshold, classification accuracy depends on the choice of threshold. Since AUC-optimized scores do not directly determine an appropriate classification threshold, improvements in AUC do not necessarily translate into improvements in classification accuracy. Investigating threshold selection and the resulting classification performance would be an interesting direction for future work.

I thank the Authors for clarification and changes to the text which now make it clear that the posteriors are available for labeled and not positive observations. Thus I obviously agree that \pi may not be estimated by simple averaging. Still, I believe that the studied model has very limited applicability and additional information makes the problem much easier. It is also obvious that it is sufficient to have increasing transformation of posteriors instead of exact posteriors but its only because Authors study AUC maximisation, this will not work e.g. for accuracy maximisation.
I also thank the Authors for the information that they have adapted case-control designed uPU and nnPU procedures to one-sample setting. However, this does not change the fact that those are designed for SCAR scenario. As I wrote in my original appraisal, comparison with other SAR-designed methods like VAE-PU or JERM would be much more informative (although still not fully enlightening as those methods do not avail themselves of additional posteriori values). I keep my score.

Reviewer_E9E4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies AUC maximization from positive-unlabeled (PU) data under the SAR setting, where the labeling probability depends on features. The proposed method uses positive-confidence scores and a labeled-vs-unlabeled classifier to estimate propensity weights and correct selection bias.

Critical strengths

The paper is well organized. The problem is clear, the proposed methods are well explained and the extensive experiments are conducted.

Critical weaknesses
  1. The novelty is limited. The main components, such as propensity estimation have been studied before. The main contribution appears to be using positive-confidence to estimate propensity scores.
  2. The confidence classifier is trained on 10,000 fully labeled instances, while the PU training set has only 5,000 instances. Since this classifier could directly perform AUC ranking, its test AUC should be reported. PN AUC training on the additional labeled data is also a missing baseline.
  3. Proposed methods require additional confidence classifier while other PU methods such as PUAUC do not need. In experiment results (Table 1), the performance of proposed methods gain minor improvement over PUAUC. When using human annotators as the confidence score, the proposed method even underperform w/o Conf method on some task, which suggests that the method is heavily relies on high-quality confidence score. However, human annotators can be noisy and training a confience classifier required a large amount of labeled data, which limits the applicability of the method to real-world tasks.
Questions for authors
  1. The confidence classifier is trained on 10,000 fully labeled instances, which is twice the size of the PU training set. Could the authors study the impact of confidence classifier's training set size on the final PU training performance?
  2. Is there any more data-efficient way to obtain the confidence score?
  3. Can the proposed method be extended to Partial AUC maximization problem?
3 comments

We appreciate the reviewer's thoughtful comments and valuable suggestions.

  1. The novelty is limited. The main components, such as propensity estimation have been studied before. The main contribution appears to be using positive-confidence to estimate propensity scores.

We would like to further clarify the contributions of our work. Our primary contribution is to derive a pairwise AUC risk reformulation that enables AUC maximization from biased PU data equipped with positive-confidence under the SAR setting. Existing PU-based AUC maximization methods assume SCAR, while existing SAR-based PU learning methods focus on classification without utilizing positive-confidence.

Furthermore, Proposition 4.1 establishes that our objective induces the Bayes-optimal AUC ranking even when the available confidence is an arbitrary strictly increasing transformation of the true posterior probability. To the best of our knowledge, such a theoretical guarantee has not been established in existing confidence-based learning methods, which typically assume that confidence represents the true posterior probability. Notably, our guarantee holds even when the resulting weights do not recover the true inverse propensity scores. The empirical robustness to these transformations is also demonstrated in Figure 2 and Appendix F.7.

We also provide theoretical analyses of the sensitivity to labeling probability estimation errors and the generalization behavior of the proposed estimator (Appendix B). Taken together, these contributions provide meaningful novelty beyond existing methods.

  1. The confidence classifier is trained on 10,000 fully labeled instances, while the PU training set has only 5,000 instances. Since this classifier could directly perform AUC ranking, its test AUC should be reported.

Thank you for this comment. We would like to clarify the purpose of the additional 10,000 labeled instances. They were used solely to train a probabilistic classifier for generating high-quality confidence values in a controlled evaluation of our proposed AUC risk estimator, following the evaluation protocol adopted in previous confidence-based learning studies.

As requested, we evaluated this classifier (Teacher) by directly using its predicted probabilities as ranking scores. The test AUCs are reported below.

DatasetOursTeacher
MNIST0.89790.9494
FMNIST0.94950.9886
SVHN0.66730.5274
CIFAR100.86430.8453
Diabetes0.73420.7977
Blood0.55090.6406

The Teacher achieved higher AUCs on four datasets, while our method outperformed it on SVHN and CIFAR10.

Importantly, directly using the Teacher corresponds to a different learning setting with substantially stronger supervision, where a fully supervised classifier capable of scoring arbitrary test instances is available, whereas our problem setting assumes confidence values only for biased labeled positives data. Thus, the Teacher's AUC serves as a useful reference rather than a like-for-like comparison with our method.

Our main contribution is not to outperform fully supervised learning with abundant labeled data, but to establish an AUC maximization framework for biased PU data equipped with confidence. We also evaluated our method using confidence obtained from real-world human annotators (Table 2), without training an additional confidence classifier.

  1. Proposed methods require additional confidence classifier while other PU methods such as PUAUC do not need. and > 2. Is there any more data-efficient way to obtain the confidence score?

Thank you for raising this concern. We would like to clarify that our method requires positive-confidence values for biased labeled positive instances, but does not require training an additional confidence classifier. The classifier trained on 10,000 fully labeled instances was used to generate high-quality confidence values for controlled evaluation, following the standard evaluation protocol in previous confidence-based learning studies.

As discussed in the Introduction, positive-confidence can be obtained from existing annotation or operational processes. For example, human annotators can provide confidence scores when assigning positive labels, or confidence can be obtained from existing annotation results or pretrained classifiers. Thus, our method does not necessarily require collecting an additional large fully labeled dataset.

In experiment results (Table 1), the performance of proposed methods gain minor improvement over PUAUC. When using human annotators as the confidence score, the proposed method even underperform w/o Conf method on some task.

Thank you for raising these concerns. As shown in Table 1, our method outperformed PUAUC on five of the six datasets and achieved the best or statistically comparable performance among the evaluated methods on all six datasets.

To more directly evaluate the effectiveness of our method in correcting selection bias, we conducted an additional subgroup analysis focusing on low-propensity positive instances. Since these instances are underrepresented among labeled positives, their ranking performance may be particularly affected by uncorrected selection bias.

Following the original experimental settings (Appendices E.1 and E.2), 90% of labeled positive training instances were sampled from high-propensity subgroups and 10% from low-propensity subgroups. We evaluated test AUC separately for each positive subgroup using the corresponding positive test instances and all negative test instances. We also included CIFAR10-H and FMNIST-H with human-derived confidence. The following tables report average test AUCs.

Low-propensity positive subgroups

DatasetOursw/oConfPUAUC
MNIST0.84260.79010.7901
FMNIST0.89230.86270.8680
SVHN0.58720.59220.5926
CIFAR100.86330.83180.8573
Diabetes0.62510.56010.5947
Blood0.40960.37700.3839
CIFAR10-H0.60100.60190.5699
FMNIST-H0.72520.71480.6988

High-propensity positive subgroups

DatasetOursw/oConfPUAUC
MNIST0.93470.93470.9446
FMNIST0.98710.98610.9890
SVHN0.69500.69230.6973
CIFAR100.86540.84250.8666
Diabetes0.84860.83180.8457
Blood0.69460.68350.6976
CIFAR10-H0.59730.59780.5927
FMNIST-H0.89580.88710.9196

For low-propensity subgroups, our method outperformed PUAUC on seven of the eight datasets and w/oConf on six datasets. Although improvements were less consistent for high-propensity subgroups, these results suggest that our method can be particularly beneficial for positive instances underrepresented in the labeled data, with benefits that may be less apparent from overall AUC alone.

Regarding human-derived confidence, our method slightly underperformed w/oConf on CIFAR10-H in overall AUC (0.5991 vs. 0.5998), but outperformed it on FMNIST-H (0.8285 vs. 0.8192). Moreover, on FMNIST-H, our method significantly outperformed both w/oConf and PUAUC for the low-propensity subgroup. We will include these additional subgroup results in the revised paper.

The method is heavily relies on high-quality confidence score. However, human annotators can be noisy, which limits the applicability of the method to real-world tasks.

Thank you for raising this important concern. We agree that the usefulness of positive-confidence depends on its informativeness. However, the limited improvement on CIFAR10-H does not necessarily indicate sensitivity to annotation noise. As discussed in Section 5.3, confidence values on this dataset were close to one, providing limited additional information.

Importantly, our method does not require perfectly calibrated confidence. Proposition 4.1 guarantees the Bayes-optimal AUC ranking when confidence is any strictly increasing transformation of the true posterior probability. Although this guarantee does not cover arbitrary noisy confidence, Appendix F.8 demonstrates empirical robustness to certain levels of additive Gaussian noise.

These results suggest that informative confidence can be beneficial even when it is imperfect, although robustness to arbitrary confidence noise is not guaranteed. We will clarify this limitation in the revised paper.

  1. Can the proposed method be extended to Partial AUC maximization problem?

Thank you for this interesting question. We believe that our confidence-based correction of biased positive data could also be useful for partial AUC (pAUC) maximization. However, unlike full AUC, pAUC involves score thresholds determined by the negative-score distribution, which is not directly available in our setting. Therefore, extending our method to pAUC maximization under the SAR setting requires additional theoretical investigation. We consider this an interesting direction for future work.

Thank you again for your careful review and constructive feedback.

As the author–reviewer discussion period is coming to a close, we would appreciate your consideration of our responses and the revisions made to the paper.

Please let us know if you have any further questions or concerns.

Reviewer_bTU8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on AUC maximization from selection-biased positive-unlabeled data with confidence scores on labeled positives. It derives a weighted risk estimator that avoids explicit class-prior estimation and establishes population-level ranking invariance under strictly increasing transformations of posterior confidence. The experiments cover six datasets with classifier-generated confidence and two with human-derived confidence, using constructed selection biases and sensitivity analyses. The results show the effectiveness of the proposed methods.

Critical strengths
  1. The risk rewriting explicitly accounts for biased positive selection while avoiding class-prior estimation. This provides a principled extension of PU-AUC learning beyond SCAR.
  2. The method can tolerate systematic confidence miscalibration under a strict monotonicity condition. This meaningfully broadens its theoretical scope beyond requiring exact posterior probabilities.
Critical weaknesses
  1. The additional supervision assumption is insufficiently connected to practice. Posterior confidence, or a common strictly increasing transformation thereof, is substantially more informative than reliable positive labels or generic confidence scores. Annotator agreement and operational scores need not satisfy this relationship. This weakens the central claim that the method addresses realistic SAR settings without restrictive assumptions.
  2. The experiments do not establish the source of the gains. Confidence is generated using 10,000 additional fully labeled examples, while the PU learner receives only 25–100 labeled positives. The teacher’s AUC and simple confidence-transfer baselines are missing. Moreover, w/o Conf removes confidence rather than isolating selection correction. Consequently, the contribution of the proposed correction cannot be separated from that of additional supervision.
  3. Effective debiasing is not directly demonstrated. The ranking guarantee assumes the true labeling probability, whereas implementation uses an estimated and clipped denominator. Weight distributions do not establish recovery of inverse propensity weights, and strong clipping remains competitive (Table 9). Without recovery diagnostics or subgroup results, it remains unclear whether the observed improvements validate the paper’s central debiasing mechanism.
Questions for authors
  1. What evidence supports the required relationship between human-derived confidence and the target posterior? Is the intended applicability restricted to scores satisfying this structural assumption?
  2. How does the method compare with the confidence teacher and with confidence weighting without the inverse-labeling-probability factor? These comparisons would clarify whether the proposed correction provides additional value.
  3. What fraction of denominators is clipped, and is there evidence that the estimated weights recover the known selection mechanism or improve performance on low-propensity positive subgroups?
3 comments

We appreciate the reviewer's thoughtful comments and valuable suggestions.

  1. The additional supervision assumption is insufficiently connected to practice. Posterior confidence, or a common strictly increasing transformation thereof, is substantially more informative than reliable positive labels or generic confidence scores. This weakens the central claim that the method addresses realistic SAR settings without restrictive assumptions.

Thank you for raising this important concern. We agree that posterior confidence provides additional supervision compared with standard PU learning and may not be available in all applications. However, our method requires confidence only for biased labeled positive instances, rather than for arbitrary unlabeled or test instances. As discussed in the Introduction, such confidence-like information can be obtained from existing annotation or operational processes in some applications.

Importantly, the strictly increasing relationship in Proposition 4.1 is a sufficient condition for guaranteeing the Bayes-optimal AUC ranking, rather than a requirement for applying our method. Although real-world confidence may not satisfy this condition exactly, Table 2 demonstrates the empirical applicability of our method using confidence obtained from real-world human annotators. Furthermore, Appendix F.8 provides empirical evidence that our method can tolerate certain levels of additive noise in confidence values. These results support the practical applicability of our method beyond the idealized confidence setting, while the theoretical guarantee remains limited to confidence satisfying the stated condition.

Finally, our claim of avoiding restrictive assumptions refers specifically to assumptions on data distributions or labeling mechanisms imposed by existing SAR-based methods, rather than the absence of assumptions on confidence. We will clarify this distinction and the scope of our theoretical guarantee in the revised paper.

  1. The experiments do not establish the source of the gains. Confidence is generated using 10,000 additional fully labeled examples, while the PU learner receives only 25–100 labeled positives. The teacher’s AUC and simple confidence-transfer baselines are missing. The contribution of the proposed correction cannot be separated from that of additional supervision.

Thank you for this important comment. We agree that it is important to distinguish the contribution of additional confidence information from that of the proposed selection correction.

First, the additional 10,000 fully labeled instances were used solely to train a probabilistic classifier (Teacher) for generating confidence values in our controlled evaluation, following previous confidence-based learning studies. As requested, we evaluated the Teacher by directly using its predicted probabilities as ranking scores.

DatasetOursTeacher
MNIST0.89790.9494
FMNIST0.94950.9886
SVHN0.66730.5274
CIFAR100.86430.8453
Diabetes0.73420.7977
Blood0.55090.6406

Although the additional labeled data indirectly provide supervision through the generated confidence values, directly using the Teacher corresponds to a different learning setting where a fully supervised classifier capable of scoring arbitrary test instances is available. In contrast, our setting assumes confidence only for biased labeled positives. Thus, the Teacher's AUC serves as a useful reference rather than a comparison under the same supervision assumptions.

Second, Table 1 already includes Pconf and NPU, two existing confidence-based learning methods. To further distinguish the contribution of selection correction from that of additional confidence information, we conducted an additional ablation study (Conf-only) while keeping the available confidence unchanged. As detailed in our next response, the results provide evidence that the proposed selection correction can improve ranking performance for underrepresented positive instances beyond simply using confidence information.

How does the method compare with confidence weighting without the inverse-labeling-probability factor? These comparisons would clarify whether the proposed correction provides additional value. Without subgroup results, it remains unclear whether the observed improvements validate the paper’s central debiasing mechanism.

Thank you for the suggestions. To isolate the contribution of selection correction, we conducted an additional ablation study by replacing the proposed weight r(x)/u^(x)r(\mathbf{x})/\hat{u}(\mathbf{x}) in Eq. (17) with r(x)r(\mathbf{x}) alone (Conf-only). We further evaluated AUC separately for high- and low-propensity positive subgroups to investigate whether the proposed correction improves ranking performance for underrepresented positive instances.

We used the same experimental settings and selection mechanisms as in the original experiments (Appendices E.1 and E.2), where 90% of labeled positives were sampled from high-propensity subgroups and 10% from low-propensity subgroups. Each subgroup AUC was calculated using the corresponding positive test instances and all negative test instances. We also included CIFAR10-H and FMNIST-H, which use confidence obtained from real-world human annotators.

The following tables report average test AUCs.

Low-propensity positive subgroups

DatasetOursConf-onlyPUAUC
MNIST0.84260.83190.7901
FMNIST0.89230.88250.8680
SVHN0.58720.57940.5926
CIFAR100.86330.86180.8573
Diabetes0.62510.63410.5947
Blood0.40960.41170.3839
CIFAR10-H0.60100.57020.5699
FMNIST-H0.72520.70800.6988

High-propensity positive subgroups

DatasetOursConf-onlyPUAUC
MNIST0.93470.94350.9446
FMNIST0.98710.98670.9890
SVHN0.69500.68180.6973
CIFAR100.86540.86210.8666
Diabetes0.84860.85120.8457
Blood0.69460.70360.6976
CIFAR10-H0.59730.59240.5927
FMNIST-H0.89580.92170.9196

For low-propensity subgroups, our method outperformed Conf-only on six of the eight datasets and PUAUC on seven of the eight datasets. Notably, on both datasets with human-derived confidence (CIFAR10-H and FMNIST-H), our method significantly outperformed both baselines for low-propensity subgroups, while improvements were less consistent for high-propensity subgroups.

These results demonstrate that the proposed correction can provide additional value beyond confidence weighting alone, particularly for low-propensity positive subgroups, thereby providing empirical support for the proposed debiasing mechanism. We will include these additional results in the revised paper.

The ranking guarantee assumes the true labeling probability, whereas implementation uses an estimated and clipped denominator. Weight distributions do not establish recovery of inverse propensity weights, and strong clipping remains competitive (Table 9).

Thank you for these comments. We acknowledge that the ranking guarantee assumes the true labeling probability, whereas our implementation uses an estimated and clipped value. Appendix B.2 analyzes the sensitivity to estimation errors in the labeling probability. In practice, clipping is used to prevent excessively large weights and stabilize training.

Regarding Table 9, when u^(x)\hat{u}(\mathbf{x}) is below the clipping threshold τ\tau, the proposed weight becomes r(x)/τr(\mathbf{x})/\tau, which is proportional to confidence-only weighting. Thus, strong clipping makes the proposed method closer to Conf-only, consistent with our additional ablation results showing relatively small differences in overall AUC.

However, reducing the clipping threshold from 0.1 to 0.01 improved test AUC for low-propensity positive subgroups on six of the eight datasets, with statistically significant improvements on three datasets. In particular, on both datasets with human-derived confidence, this AUC significantly improved from 0.5746 to 0.6010 on CIFAR10-H and from 0.7153 to 0.7252 on FMNIST-H. These results provide further empirical support for the benefits of selection correction for underrepresented positive instances.

  1. What evidence supports the required relationship between human-derived confidence and the target posterior? Is the intended applicability restricted to scores satisfying this structural assumption?

As clarified in our response to Weakness 1, the strictly increasing relationship is a sufficient condition for the theoretical ranking guarantee, rather than a requirement for applying our method. We do not claim that human-derived confidence necessarily satisfies this condition. Nevertheless, Table 2 and Appendix F.8 provide empirical evidence of applicability and robustness beyond the idealized confidence setting.

Thank you again for your careful review and constructive feedback.

As the author–reviewer discussion period is coming to a close, we would appreciate your consideration of our responses and the revisions made to the paper.

Please let us know if you have any further questions or concerns.

Comments on the paper

1 comment

We have uploaded a revised version of our paper incorporating clarifications and additional experimental results based on the reviewer feedback and our rebuttal responses.

The revisions include clarifications of the one-sample scenario, the scope of our theoretical guarantee, and the experimental settings. We have also added subgroup AUC comparisons and an additional ablation study (Appendix F.12), the Teacher's AUC results (Appendix F.13), and further discussions of clipping and noisy confidence (Appendices F.9 and F.8).

All changes are highlighted in blue for ease of reference.

We appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.