- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a simple and practically valuable approach to Biased Positive-Unlabeled (PU) Learning that leverages a small set of fully labeled calibration samples to avoid explicit propensity score estimation. The paper presents a clear research motivation, supported by theoretical analysis and empirical comparisons. However, the proposed method relies heavily on relatively strong assumptions, and its methodological novelty primarily lies in integrating existing assumptions into the PU learning framework. Furthermore, the additional supervision introduced by the fully labeled calibration data has not been adequately evaluated. More comprehensive experimental comparisons with recent state-of-the-art PU learning methods are also needed to better establish the effectiveness and competitiveness of the proposed approach.
Critical strengths
-
The research problem is practically significant, and the proposed method is computationally efficient. Biased PU Learning is an important and challenging problem. By reformulating complex propensity score estimation as a one-dimensional probability calibration problem, this paper offers a relatively simple and computationally efficient solution.
-
The method is supported by theoretical analysis and experimental validation. Beyond introducing a specific calibration approach, the paper establishes a theoretical relationship between the non-traditional classifier (NTC) outputs and the true class posterior probabilities through formal theorems. It also analyzes the estimation error of the proposed method, providing theoretical support for its effectiveness.
Critical weaknesses
The lack of comparisons with recent state-of-the-art PU learning methods is the most critical weakness of this paper.
-
The methodological novelty is unclear. Compared with existing methods that are also based on FOP assumption, what are the unique contributions of this paper?
-
The benefits of additional fully labeled data have not been adequately disentangled from those of the proposed method. It remains unclear whether PUCal's performance improvements primarily stem from the proposed calibration framework or from the additional supervision provided by the fully labeled data. Furthermore, the paper lacks comparisons with recent PU learning methods that also leverage additional fully labeled data, such as Positive-Unlabeled Learning by Latent Group-Aware Meta Disambiguation.
-
Lack of comparisons with recent state-of-the-art PU learning methods. The paper lacks comparisons with several representative PU learning methods that have made important contributions to the advancement of PU learning, such as Noisy-Pair Robust Representation Alignment for Positive-Unlabeled Learning, Positive-Unlabeled Learning With Label Distribution Alignment, Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training, and Positive-Unlabeled Learning with Non-Negative Risk Estimator. Although these methods were not specifically designed for Biased PU learning, including them as baselines would help provide a more comprehensive evaluation and better demonstrate the effectiveness and competitiveness of the proposed approach.
-
Additional evaluation metrics are recommended. The authors are encouraged to include commonly used classification metrics in PU learning, such as F1-score, precision, and recall, to provide a more comprehensive evaluation of the proposed method.
Questions for authors
Please refer to the Critical Weaknesses section for details. I would be willing to raise my score if the experiments suggested in the Critical Weaknesses section are conducted.
3 comments
Q: Additional evaluation metrics are recommended. The authors are encouraged to include commonly used classification metrics in PU learning, such as F1-score, precision, and recall, to provide a more comprehensive evaluation of the proposed method.
Given the theoretical results and the construction of the method itself, our main objective was to estimate the posterior probability by transforming the probability estimated using NTC. Therefore, the most important aspect for us was the analysis of the AE (Approximation Error) metric, which allows us to assess how accurately we recover the probability , followed by the ECE (Expected Calibration Error). Metrics related to the quality of the decision rule are of secondary importance in this context. We used Balanced Accuracy as a representative metric from this group. Additional experiments for selected settings suggest that the advantage is also evident for metrics such as F1-score.
I thank the authors for their thoughtful responses, which have partially addressed my concerns. I acknowledge the differences in assumptions between biased PU learning and conventional PU learning. However, the authors have not sufficiently justified why all the suggested methods are unsuitable for comparison or adaptation to their setting. Furthermore, the experimental comparisons with recent state-of-the-art PU learning methods that I previously requested have not been provided. Therefore, I will maintain my original score.
Q: The methodological novelty is unclear. Compared with existing methods that are also based on FOP assumption, what are the unique contributions of this paper?
The difference is fundamental. Previous works based on FOP used the estimator s(x)=p(s=1∣x) and aimed to estimate the classification threshold. In this work, our objective is to estimate the true posterior probability y(x)=p(y=1∣x), which is a considerably more challenging task. More precisely, in previous works, it has been shown [1] that when the true classification function is linear and the feature distribution is elliptically contoured, the NTC classifier enables the recovery of the direction of the linear decision boundary separating the classes, with only the intercept requiring estimation.
In our setting, optimizing the threshold on a calibration set for NTC may improve the classification rule (as measured by Balanced Accuracy), but it does not improve the quality of posterior probability estimation (as measured by AE and ECE). This is clearly illustrated by the results presented in our response to Reviewer USUp.
[1] P. Teisseyre, T. Martens, J. Bekker, and J. Davis. Learning from biased positive-unlabeled data via threshold calibration. In Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, AISTATS’25, pages 2314–2322, 2025.
Q: The benefits of additional fully labeled data have not been adequately disentangled from those of the proposed method. It remains unclear whether PUCal's performance improvements primarily stem from the proposed calibration framework or from the additional supervision provided by the fully labeled data. Furthermore, the paper lacks comparisons with recent PU learning methods that also leverage additional fully labeled data, such as Positive-Unlabeled Learning by Latent Group-Aware Meta Disambiguation.
We refer to Tables 2 and 3 above (responses to Reviewer USUp) which cleary show that in terms of AE and ECE improvement is due to calibration and not to the additional supervision, which was taken adventage of by NTC-tuned.
LaGAM method invoked by the Reviewer, relies on observability of samples from positive distribution and unlabelled samples generated from marginal distribution (case-control setting), which is not the case in our setup. In contrast, we consider the single-sample setting for biased PU learning.
Q: Lack of comparisons with recent state-of-the-art PU learning methods. The paper lacks comparisons with several representative PU learning methods that have made important contributions to the advancement of PU learning, such as Noisy-Pair Robust Representation Alignment for Positive-Unlabeled Learning, Positive-Unlabeled Learning With Label Distribution Alignment, Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training, and Positive-Unlabeled Learning with Non-Negative Risk Estimator. Although these methods were not specifically designed for Biased PU learning, including them as baselines would help provide a more comprehensive evaluation and better demonstrate the effectiveness and competitiveness of the proposed approach.
We have compared our method with practically all state-of-the-art PU methods built for biased PU under single sample scenario. The methods listed by the Reviewer are designed for different scenarios, e.g. nnPU method concerns unbiased PU case -control scenario when sample for positive class is observable. Their usage in our case would result in inconsistent estimators of posterior distribution of positive class. We believe that it would be misleading to compare our method with methods built under different premises as was discussed e.g. in [1,2,3].
[1] J. Bekker and J. Davis. Learning from positive and unlabeled data: a survey. Machine Learning, 109:719–760, 2020
[2] J. Mielniczuk and A. Wawrzenczyk. Single-sample versus case-control sampling scheme for Positive Unlabeled data: the story of two scenarios. Fundamenta Informaticae, 191:1–17, 2024
[3] Wang, W., Wu, D.-D., Li, M., Zhang, J., Niu, G., and Sugiyama, M. (2026). Accessible, realistic, and fair evaluation of positive-unlabeled learning algorithms. Proceedings of the International Conference on Learning Representations (ICLR 2026)