- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method for maximizing partial AUC from positive and unlabeled data by reformulating the pAUC and its FPR dependent thresholds using positive and marginal distributions. It derives an empirical estimator, introduces a smoothed objective for classifier training, and establishes finite sample estimation bounds. Experiments on ten datasets show competitive performance against existing PU learning, AUC maximization, and pAUC maximization methods. However, the theoretical assumptions and the accuracy of threshold estimation in the low FPR regime require further examination.
Critical strengths
-
The paper addresses the important problem of pAUC maximization without labeled negative data. Its reformulation avoids the heuristic pseudo negative assumption used by an existing PU based pAUC method.
-
The paper provides a finite sample analysis of threshold estimation and pAUC estimation, including the effect of class prior estimation errors.
-
The experiments cover multiple image and tabular datasets, different numbers of labeled positives, different class priors, and alternative surrogate functions.
Critical weaknesses
-
The main methodological contribution is the reformulation of pAUC and its FPR-dependent thresholds using the positive and marginal distributions. However, the subsequent learning procedure relies on a smoothed pairwise ranking objective and score-based interval selection, which are also used in prior pAUC maximization methods. Although the proposed formulation avoids the heuristic pseudo-negative selection used by existing PU-based pAUC methods, the paper does not sufficiently clarify the methodological novelty beyond this difference. What are the substantive technical advances over existing PU risk rewriting and pAUC maximization approaches, and how do they contribute to the observed performance gains?
-
The theoretical analysis relies on the assumption that the negative score distribution has no point masses, which ensures the continuity of the FPR and supports the threshold estimation arguments. However, the applicability of this assumption to the evaluated image datasets, where tied scores may occur, is not sufficiently discussed. Moreover, the finite-sample analysis primarily establishes estimation error bounds rather than guarantees on the excess pAUC of the classifier obtained through the proposed optimization procedure. How can the authors justify or relax the no-point-masses assumption, and what guarantees can be established for the actual learning procedure beyond estimator consistency?
-
The threshold analysis in Appendix C.7 raises concerns about whether the proposed method accurately optimizes the intended low-FPR region. For example, when the target FPR is zero, the actual FPR induced by the estimated threshold reaches approximately 0.03–0.06 in several settings. Such deviations are particularly concerning when labeled positives are scarce, as they may cause the effective training interval to differ substantially from the intended interval. Can the authors quantify how these threshold estimation errors affect the final test pAUC and demonstrate that the proposed method remains effective when the target FPR is very low?
-
Although the proposed method achieves competitive results across multiple datasets, its superiority over the strongest baselines is not consistent across all settings. The paper relies on paired t-tests to identify methods that are statistically indistinguishable from the best-performing method, but it does not discuss the potential impact of conducting multiple significance tests across numerous experimental settings. Can the authors provide further evidence of the robustness of the reported improvements, including an assessment of whether the conclusions remain unchanged after appropriate multiple-comparison correction?
Questions for authors
Please refer to the Weaknesses.
5 comments
- The theoretical analysis relies on the assumption that the negative score distribution has no point masses, which ensures the continuity of the FPR and supports the threshold estimation arguments. However, the applicability of this assumption to the evaluated image datasets, where tied scores may occur, is not sufficiently discussed. ... How can the authors justify or relax the no-point-masses assumption?
Thank you for raising these important questions.
First, we clarify that Assumption A.1 is introduced as a sufficient condition for our finite-sample theoretical analysis, rather than for the PU-based rewriting itself. Specifically, the PU-based rewriting of the pairwise expression and FPR in Eqs. (8)–(10) does not require this assumption. Assumption A.1 is used to ensure FPR continuity when bounding the errors at the estimated thresholds.
To examine the practical relevance of this assumption, we additionally analyzed negative-score ties across 480 trained models on four image datasets, using 2,500 negative test instances per model. No ties were observed in 404 of the 480 models (84.2%). For each dataset, the mean pairwise tie rate, defined as the average proportion of unordered pairs of negative test instances with identical scores across its 120 models, was below . The largest observed tied-score group contained only two instances (0.08% of the negative test set). These results indicate that score ties were extremely rare in our experiments.
Furthermore, the threshold estimation bound in Theorem A.3 can be extended to allow point masses. Let denote the maximum point mass of the negative-score distribution. Using this quantity, the bound in Theorem A.3 can be extended to
\leq\epsilon_{\mathrm{FPR}}+m_s.$$ Specifically, in the proof of Theorem A.3, Eq. (50) remains valid without Assumption A.1. Taking $t\uparrow\hat t_\gamma$ in Eq. (50), we obtain $$\mathrm{FPR}(\hat t_\gamma^-) \geq\gamma-\epsilon_{\mathrm{FPR}},$$ where $\mathrm{FPR}(\hat t_\gamma^-)$ denotes the left limit of $\mathrm{FPR}(t)$ at $\hat t_\gamma$. Since $$\mathrm{FPR}(\hat t_\gamma^-) -\mathrm{FPR}(\hat t_\gamma) =P(s(\mathbf{x}^{\rm n})=\hat t_\gamma) \leq m_s,$$ Eq. (51) can be relaxed to $$\mathrm{FPR}(\hat t_\gamma) \geq\gamma-\epsilon_{\mathrm{FPR}}-m_s.$$ Combining this with Eq. (46) yields the extended bound above, corresponding to Eq. (52). Thus, allowing small point masses introduces only an additional error term of $m_s$ into the threshold estimation bound. > 2. The finite-sample analysis primarily establishes estimation error bounds rather than guarantees on the excess pAUC of the classifier obtained through the proposed optimization procedure. What guarantees can be established for the actual learning procedure beyond estimator consistency? Theorem A.6 establishes a uniform pAUC estimation error bound over the score-function class. Consequently, an exact maximizer of the empirical pAUC in Eq. (14) satisfies an excess pAUC bound of $2B_{N,N^p}$ relative to the best score function in the class. However, our actual learning procedure (Algorithm 1) uses the smoothed objective in Eq. (15) and mini-batch SGD. Extending the guarantee to this procedure requires controlling the surrogate approximation and optimization errors, which are not bounded by our current analysis. We will clarify this distinction and the corresponding limitation in the revised paper.
- The threshold analysis in Appendix C.7 raises concerns about whether the proposed method accurately optimizes the intended low-FPR region. ... Can the authors quantify how these threshold estimation errors affect the final test pAUC and demonstrate that the proposed method remains effective when the target FPR is very low?
Thank you for the insightful comment. The FPR boundary at corresponds to the extreme tail of the negative-score distribution, where accurate threshold estimation is particularly challenging with finite data. This explains why the deviations at tend to be larger than those at or . As shown in Table 9, increasing the number of labeled positive instances from 30 to 100 consistently reduces these deviations.
Regarding the effect of threshold estimation errors, Eq. (91) bounds the error in the pAUC objective caused by replacing the population thresholds with their estimated values, for a fixed score function, by Theorem A.6 further combines this threshold estimation error with the empirical pAUC estimation error.
Importantly, the test pAUCs in Tables 1 and 2 are evaluated using the true positive and negative test labeled data over the specified FPR ranges, rather than the estimated FPR thresholds. Despite the threshold estimation errors, our method achieved the highest mean test pAUC in 24 out of 32 settings and was best or not significantly different from the best in 31 out of 32 settings (Tables 1 and 2). These results demonstrate that our method remains empirically effective for maximizing the target pAUC even with finite-sample threshold estimation errors. Furthermore, the ablation study in Tables A and B (our response to Question 1) provides additional evidence of the importance of PU-based threshold estimation. With the same PU-rewritten pairwise risk, our method achieved higher mean test pAUC than the variant treating unlabeled data as negative for threshold estimation in all 32 settings.
To further evaluate the effectiveness of our method in low-FPR regions, we additionally conducted experiments on a narrower FPR interval of [0, 0.05]. The mean test pAUCs are reported in the following table.
| Data | Ours | NNPU | PUAUC | PAUC | PUpAUCh | 2step | |
|---|---|---|---|---|---|---|---|
| MNIST | 0.1 | 0.5940 ± 0.0846 | 0.4392 ± 0.1272 | 0.5265 ± 0.0603 | 0.4376 ± 0.0603 | 0.5955 ± 0.0772 | 0.5334 ± 0.0621 |
| MNIST | 0.2 | 0.5424 ± 0.0845 | 0.4713 ± 0.0755 | 0.4887 ± 0.0537 | 0.3244 ± 0.0693 | 0.5320 ± 0.0751 | 0.4863 ± 0.0562 |
| FMNIST | 0.1 | 0.8979 ± 0.0315 | 0.7403 ± 0.0286 | 0.8608 ± 0.0475 | 0.6355 ± 0.0885 | 0.9040 ± 0.0293 | 0.8768 ± 0.0464 |
| FMNIST | 0.2 | 0.8819 ± 0.0445 | 0.7521 ± 0.0380 | 0.8239 ± 0.0606 | 0.4972 ± 0.1132 | 0.8894 ± 0.0334 | 0.8319 ± 0.0597 |
| SVHN | 0.1 | 0.1689 ± 0.0585 | 0.0287 ± 0.0038 | 0.1378 ± 0.0586 | 0.1161 ± 0.0259 | 0.1698 ± 0.0680 | 0.1398 ± 0.0599 |
| SVHN | 0.2 | 0.1213 ± 0.0558 | 0.0287 ± 0.0038 | 0.1092 ± 0.0421 | 0.0847 ± 0.0258 | 0.0993 ± 0.0604 | 0.0757 ± 0.0422 |
| CIFAR10 | 0.1 | 0.3390 ± 0.0817 | 0.0298 ± 0.0273 | 0.3279 ± 0.0764 | 0.2096 ± 0.0392 | 0.3280 ± 0.0717 | 0.3345 ± 0.0837 |
| CIFAR10 | 0.2 | 0.3028 ± 0.0801 | 0.0436 ± 0.0483 | 0.2931 ± 0.0689 | 0.1378 ± 0.0382 | 0.2886 ± 0.0669 | 0.2945 ± 0.0697 |
| SUSY | 0.1 | 0.3675 ± 0.0401 | 0.2780 ± 0.0783 | 0.3217 ± 0.0617 | 0.2042 ± 0.0411 | 0.3458 ± 0.0413 | 0.3183 ± 0.0624 |
| SUSY | 0.2 | 0.3356 ± 0.0458 | 0.2861 ± 0.0597 | 0.2949 ± 0.0618 | 0.1297 ± 0.0308 | 0.2868 ± 0.0495 | 0.2900 ± 0.0629 |
| CODRNA | 0.1 | 0.5837 ± 0.0603 | 0.2491 ± 0.2235 | 0.6045 ± 0.0724 | 0.4147 ± 0.0728 | 0.5877 ± 0.0597 | 0.6054 ± 0.0697 |
| CODRNA | 0.2 | 0.5627 ± 0.0530 | 0.5569 ± 0.0645 | 0.5710 ± 0.0704 | 0.3226 ± 0.0817 | 0.5620 ± 0.0526 | 0.5759 ± 0.0692 |
| IJCNN | 0.1 | 0.5326 ± 0.0989 | 0.1907 ± 0.1076 | 0.4437 ± 0.0868 | 0.3569 ± 0.0685 | 0.4838 ± 0.1342 | 0.4489 ± 0.0910 |
| IJCNN | 0.2 | 0.4195 ± 0.1017 | 0.3508 ± 0.0791 | 0.4052 ± 0.0747 | 0.2619 ± 0.0547 | 0.3013 ± 0.1772 | 0.4003 ± 0.0759 |
| EPSILON | 0.1 | 0.1183 ± 0.0327 | 0.0510 ± 0.0054 | 0.1193 ± 0.0321 | 0.0981 ± 0.0243 | 0.1173 ± 0.0304 | 0.1196 ± 0.0342 |
| EPSILON | 0.2 | 0.0979 ± 0.0303 | 0.0503 ± 0.0075 | 0.1046 ± 0.0290 | 0.0805 ± 0.0167 | 0.0942 ± 0.0274 | 0.1083 ± 0.0318 |
| Avg | 0.4291 ± 0.2490 | 0.2842 ± 0.2557 | 0.4021 ± 0.2376 | 0.2695 ± 0.1719 | 0.4116 ± 0.2584 | 0.4025 ± 0.2435 | |
| #Best | 14 | 0 | 6 | 0 | 9 | 7 |
Our method achieved the highest average test pAUC across all experimental settings (0.4291) and performed best or comparably to the best method in 14 out of 16 settings. These results further support the effectiveness of our method in low-FPR regions. We will include these results in the revised paper.
We thank the reviewer for the careful reading and constructive feedback.
Thank you for the question. We agree that the smoothed pairwise ranking loss and score-based interval selection are established techniques in existing pAUC maximization methods. However, our key technical contribution lies in the PU-based reformulation of both the pairwise risk and the FPR-dependent thresholds required for pAUC maximization.
Existing pAUC maximization methods usually require labeled negative data to determine the FPR-dependent thresholds and calculate the pairwise risk. On the other hand, existing PU risk rewriting techniques used for classification and AUC risks do not directly address the FPR-dependent thresholds. Thus, simply combining conventional pAUC maximization with PU-based pairwise risk rewriting is insufficient to recover the target pAUC objective. In Section 4, we address this issue by reformulating both components using only the positive and marginal distributions.
Furthermore, Iwata et al. (2020) heuristically treat selected unlabeled instances as negative without applying PU risk rewriting to the pairwise risk. In contrast, our method directly reformulates both the pairwise risk and the FPR-dependent thresholds, avoiding such heuristic assumptions.
To demonstrate the importance of our PU-based threshold reformulation, we conducted an additional comparison with a variant of our method without PU-based threshold estimation (Ours w/o PU thresholds). This variant uses our PU-rewritten pairwise risk in Eq. (15), but determines the thresholds from unlabeled data by simply treating them as negative data. The average test pAUCs are as follows:
Table A: Ablation results for the FPR interval [0, 0.1]
Table B: Ablation results for the FPR interval [0, 0.2]
Our method achieved higher mean test pAUC in all 32 settings, with average improvements of 0.164 and 0.080 for the FPR intervals [0, 0.1] and [0, 0.2], respectively. These results demonstrate that PU-based threshold estimation is essential for obtaining the observed performance gains and that our contribution goes beyond standard PU-based pairwise risk rewriting. We will include these results in the revised paper.