- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper concerns AUC maximisation problem for modified biased PU model (single-sample scenario) when additional sample of covariates for which posterior probability of positive class is given. The Authors use standard method of replacing theoretical AUC with AUC based on logistic function which approximates indicator function appearing in the definition of AUC for score under study and derive explicit formula for AUC in terms of observable quantities: probability distribution of positive class, of general population, posteriors of being in positive class and being labeled , and unobservable prior which does not affect maximisation of AUC. The manuscript contains the result that minimizer of the AUC does not change when posterior is transformed by increasing transformation (Proposition 4.1) and some numerical evidence that the method works (in terms of yielding larger AUC) when compared with scores obtained by other approaches, mostly based on SCAR assumption.
Critical strengths
Deriviation of explicit formula for for modified AUC in this setting is novel and may be useful.
The paper is clearly written and easy to follow.
Critical weaknesses
The main weakness is that assumption of availability of posterior values for additional samples is hardly realistic and makes the problem much easier: estimation of the posterior for positive class is the main hurdle for the PU problem.
The approach and thus experimental study is based on simple-sample scenario, whereas all considered competitors (save PG) were derived for case-control scenario, and, importantly, most of them are designed for unbiased setting. The Authors do not consider standard approaches of estimating posterior in PU setting such as e.g. SAREM, LBE, VAE-PU and JERM.
Questions for authors
- The statement of the model in (8) is confusing as labeled and unlabeled sequences are split and do not pertain to one iid sequence. In particular, formula for yields reasonable estimator of only in the setting when (x_i,o_i) iid sample is observable: here, one can freely take N:=2N for observable data, without changing any parameters of the model, but influencing asymptotic behaviour of and minimizer of AUC.
- Line 289: It seems that can be estimated in this context by simple averaging of .
- It would be informative to know how the obtained scores translate into classifier larger accuracies of corresponding classifiers. Was it attempted ?
5 comments
- Line 289: It seems that can be estimated in this context by simple averaging of .
Thank you for this comment. Indeed, the class-prior can be estimated by averaging the posterior probabilities over samples drawn from the marginal distribution . However, in our problem setting, confidence values are available only for biased labeled positive instances, as defined in Eq. (8). Therefore, simply averaging the available confidence values does not generally yield , since the labeled positive distribution differs from the marginal distribution .
Furthermore, when the observed confidence is a strictly increasing transformation of the true posterior probability, as considered in Proposition 4.1, even averaging the confidence values over the marginal distribution does not generally yield .
- It would be informative to know how the obtained scores translate into classifier larger accuracies of corresponding classifiers. Was it attempted ?
Thank you for this suggestion. We have not evaluated classification accuracy, as our work specifically focuses on AUC maximization under class imbalance. While AUC evaluates ranking performance independently of the decision threshold, classification accuracy depends on the choice of threshold. Since AUC-optimized scores do not directly determine an appropriate classification threshold, improvements in AUC do not necessarily translate into improvements in classification accuracy. Investigating threshold selection and the resulting classification performance would be an interesting direction for future work.
I thank the Authors for clarification and changes to the text which now make it clear that the posteriors are available for labeled and not positive observations. Thus I obviously agree that \pi may not be estimated by simple averaging. Still, I believe that the studied model has very limited applicability and additional information makes the problem much easier. It is also obvious that it is sufficient to have increasing transformation of posteriors instead of exact posteriors but its only because Authors study AUC maximisation, this will not work e.g. for accuracy maximisation.
I also thank the Authors for the information that they have adapted case-control designed uPU and nnPU procedures to one-sample setting. However, this does not change the fact that those are designed for SCAR scenario. As I wrote in my original appraisal, comparison with other SAR-designed methods like VAE-PU or JERM would be much more informative (although still not fully enlightening as those methods do not avail themselves of additional posteriori values). I keep my score.
We appreciate the reviewer's thoughtful comments and valuable suggestions.
Thank you for raising this important concern. We agree that estimating posterior probabilities is a fundamental challenge in standard PU learning and that positive-confidence provides additional supervision. However, our work considers a different learning setting, where confidence is available only for biased labeled positive instances, rather than for arbitrary unlabeled or test instances.
As discussed in the Introduction, such confidence can arise from existing annotation or operational processes. Furthermore, our method does not require exact posterior probabilities: Proposition 4.1 establishes that an arbitrary strictly increasing transformation of the true posterior probability is sufficient to recover the Bayes-optimal AUC ranking. We also demonstrate the applicability of our method using human-derived confidence in Table 2.
Our goal is not to solve the conventional PU learning problem, but to introduce and investigate a new weakly supervised learning setting that exploits available positive-confidence to address selection bias. We believe that studying how such additional information can be utilized for AUC maximization under SAR constitutes a meaningful research problem, distinct from conventional PU learning.
Thank you for this comment. We would like to clarify the following points.
First, although some of the comparison methods were originally developed for the case-control scenario, we explicitly adapted nnPU, PUAUC, PUSB, and NPU to the one-sample scenario by exactly rewriting expectations over the marginal distribution using the labeled-positive and unlabeled distributions, as described in Appendix E.3. Therefore, these methods were not directly applied using their original case-control objectives. We will emphasize this point in the revised paper.
Second, our comparisons are not limited to methods assuming SCAR. PUSB (Kato et al., 2019) and PG (Gerych et al., 2022) are specifically designed to address selection bias. Moreover, we already compared our method with LBE (Gong et al., 2021), a representative SAR-based PU learning method applicable to the one-sample scenario, in Appendix F.6 (Table 7). Our method outperformed LBE on all six datasets. We will make this comparison more prominent in the revised paper.
Thank you for pointing out this ambiguity. We clarify that our problem setting follows the one-sample scenario, where V=Np+N instances are initially sampled i.i.d. from the joint distribution p(x,o) and then divided into labeled positive and unlabeled data according to o.
Thus, Np and N are determined by the original sampling process and cannot be chosen arbitrarily. In particular, α^=Np/(Np+N) is the empirical proportion of instances with o=1 and is a consistent estimator of α=p(o=1) by the law of large numbers.
Doubling N alone, as suggested in the comment, would change the sampling protocol and would no longer correspond to an i.i.d. sample from the same joint distribution p(x,o).
Eq. (8) describes the conditional distributions of the two groups after splitting the original sample. We agree that this distinction was not sufficiently clear and will revise Section 4.1 to explicitly describe the original i.i.d. sampling process and the resulting sample counts.