acceptodds
Under review as a conference paper at ICLR 2027

Calibration-Enhanced Approach to Biased Positive-Unlabeled Learning

Abstract

Positive-Unlabeled (PU) learning addresses the problem of training a binary classifier using an incomplete training dataset consisting of a subset of labeled positive instances and a set of unlabeled instances, which contains both positive and negative examples. A practically relevant and recently intensively studied scenario is biased PU learning, where the labeling mechanism determining which positive examples receive labels may depend on the feature values of the instances. A naive method in PU learning, known as the non-traditional classifier (NTC), treats all unlabeled instances as negative. However, this leads to a biased estimation of the true class posterior probability. More effective learning from biased PU data is possible but challenging, as it typically requires estimating the propensity score, defined as the instance-specific probability that a positive example is labeled. Existing propensity score estimation methods are often computationally expensive, since they rely on iterative procedures that alternately update the classifier and the propensity score model. In this work, we show that under relatively weak functional assumption on the propensity score, it is possible to accurately estimate the class posterior without explicitly modeling the former, provided that a small calibration set is available. The proposed approach (called PUCal) is based on an appropriate calibration of probabilities produced by the NTC on a small set of fully observable observations. We show that such a procedure leads to a more accurate classifier compared to existing methods for biased PU learning even when only a very limited number of observations are available in the calibration set. We also provide estimation error bounds for the proposed method.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Positive-Unlabeled Learning, PU learning, biased PU learning, propensity score, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dEraOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a simple and practically valuable approach to Biased Positive-Unlabeled (PU) Learning that leverages a small set of fully labeled calibration samples to avoid explicit propensity score estimation. The paper presents a clear research motivation, supported by theoretical analysis and empirical comparisons. However, the proposed method relies heavily on relatively strong assumptions, and its methodological novelty primarily lies in integrating existing assumptions into the PU learning framework. Furthermore, the additional supervision introduced by the fully labeled calibration data has not been adequately evaluated. More comprehensive experimental comparisons with recent state-of-the-art PU learning methods are also needed to better establish the effectiveness and competitiveness of the proposed approach.

Critical strengths
  1. The research problem is practically significant, and the proposed method is computationally efficient. Biased PU Learning is an important and challenging problem. By reformulating complex propensity score estimation as a one-dimensional probability calibration problem, this paper offers a relatively simple and computationally efficient solution.

  2. The method is supported by theoretical analysis and experimental validation. Beyond introducing a specific calibration approach, the paper establishes a theoretical relationship between the non-traditional classifier (NTC) outputs and the true class posterior probabilities through formal theorems. It also analyzes the estimation error of the proposed method, providing theoretical support for its effectiveness.

Critical weaknesses

The lack of comparisons with recent state-of-the-art PU learning methods is the most critical weakness of this paper.

  1. The methodological novelty is unclear. Compared with existing methods that are also based on FOP assumption, what are the unique contributions of this paper?

  2. The benefits of additional fully labeled data have not been adequately disentangled from those of the proposed method. It remains unclear whether PUCal's performance improvements primarily stem from the proposed calibration framework or from the additional supervision provided by the fully labeled data. Furthermore, the paper lacks comparisons with recent PU learning methods that also leverage additional fully labeled data, such as Positive-Unlabeled Learning by Latent Group-Aware Meta Disambiguation.

  3. Lack of comparisons with recent state-of-the-art PU learning methods. The paper lacks comparisons with several representative PU learning methods that have made important contributions to the advancement of PU learning, such as Noisy-Pair Robust Representation Alignment for Positive-Unlabeled Learning, Positive-Unlabeled Learning With Label Distribution Alignment, Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training, and Positive-Unlabeled Learning with Non-Negative Risk Estimator. Although these methods were not specifically designed for Biased PU learning, including them as baselines would help provide a more comprehensive evaluation and better demonstrate the effectiveness and competitiveness of the proposed approach.

  4. Additional evaluation metrics are recommended. The authors are encouraged to include commonly used classification metrics in PU learning, such as F1-score, precision, and recall, to provide a more comprehensive evaluation of the proposed method.

Questions for authors

Please refer to the Critical Weaknesses section for details. I would be willing to raise my score if the experiments suggested in the Critical Weaknesses section are conducted.

3 comments

Q: The methodological novelty is unclear. Compared with existing methods that are also based on FOP assumption, what are the unique contributions of this paper?

The difference is fundamental. Previous works based on FOP used the estimator s(x)=p(s=1∣x)s(x)=p(s=1|x) and aimed to estimate the classification threshold. In this work, our objective is to estimate the true posterior probability y(x)=p(y=1∣x)y(x)=p(y=1|x), which is a considerably more challenging task. More precisely, in previous works, it has been shown [1] that when the true classification function is linear and the feature distribution is elliptically contoured, the NTC classifier enables the recovery of the direction of the linear decision boundary separating the classes, with only the intercept requiring estimation.

In our setting, optimizing the threshold on a calibration set for NTC may improve the classification rule (as measured by Balanced Accuracy), but it does not improve the quality of posterior probability estimation (as measured by AE and ECE). This is clearly illustrated by the results presented in our response to Reviewer USUp.

[1] P. Teisseyre, T. Martens, J. Bekker, and J. Davis. Learning from biased positive-unlabeled data via threshold calibration. In Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, AISTATS’25, pages 2314–2322, 2025.

Q: The benefits of additional fully labeled data have not been adequately disentangled from those of the proposed method. It remains unclear whether PUCal's performance improvements primarily stem from the proposed calibration framework or from the additional supervision provided by the fully labeled data. Furthermore, the paper lacks comparisons with recent PU learning methods that also leverage additional fully labeled data, such as Positive-Unlabeled Learning by Latent Group-Aware Meta Disambiguation.

We refer to Tables 2 and 3 above (responses to Reviewer USUp) which cleary show that in terms of AE and ECE improvement is due to calibration and not to the additional supervision, which was taken adventage of by NTC-tuned.

LaGAM method invoked by the Reviewer, relies on observability of samples from positive distribution and unlabelled samples generated from marginal distribution (case-control setting), which is not the case in our setup. In contrast, we consider the single-sample setting for biased PU learning.

Q: Lack of comparisons with recent state-of-the-art PU learning methods. The paper lacks comparisons with several representative PU learning methods that have made important contributions to the advancement of PU learning, such as Noisy-Pair Robust Representation Alignment for Positive-Unlabeled Learning, Positive-Unlabeled Learning With Label Distribution Alignment, Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training, and Positive-Unlabeled Learning with Non-Negative Risk Estimator. Although these methods were not specifically designed for Biased PU learning, including them as baselines would help provide a more comprehensive evaluation and better demonstrate the effectiveness and competitiveness of the proposed approach.

We have compared our method with practically all state-of-the-art PU methods built for biased PU under single sample scenario. The methods listed by the Reviewer are designed for different scenarios, e.g. nnPU method concerns unbiased PU case -control scenario when sample for positive class is observable. Their usage in our case would result in inconsistent estimators of posterior distribution of positive class. We believe that it would be misleading to compare our method with methods built under different premises as was discussed e.g. in [1,2,3].

[1] J. Bekker and J. Davis. Learning from positive and unlabeled data: a survey. Machine Learning, 109:719–760, 2020

[2] J. Mielniczuk and A. Wawrzenczyk. Single-sample versus case-control sampling scheme for Positive Unlabeled data: the story of two scenarios. Fundamenta Informaticae, 191:1–17, 2024

[3] Wang, W., Wu, D.-D., Li, M., Zhang, J., Niu, G., and Sugiyama, M. (2026). Accessible, realistic, and fair evaluation of positive-unlabeled learning algorithms. Proceedings of the International Conference on Learning Representations (ICLR 2026)

Q: Additional evaluation metrics are recommended. The authors are encouraged to include commonly used classification metrics in PU learning, such as F1-score, precision, and recall, to provide a more comprehensive evaluation of the proposed method.

Given the theoretical results and the construction of the method itself, our main objective was to estimate the posterior probability p(y=1∣x)p(y=1|x) by transforming the probability p(s=1∣x)p(s=1|x) estimated using NTC. Therefore, the most important aspect for us was the analysis of the AE (Approximation Error) metric, which allows us to assess how accurately we recover the probability p(y=1∣x)p(y=1|x), followed by the ECE (Expected Calibration Error). Metrics related to the quality of the decision rule are of secondary importance in this context. We used Balanced Accuracy as a representative metric from this group. Additional experiments for selected settings suggest that the advantage is also evident for metrics such as F1-score.

I thank the authors for their thoughtful responses, which have partially addressed my concerns. I acknowledge the differences in assumptions between biased PU learning and conventional PU learning. However, the authors have not sufficiently justified why all the suggested methods are unsuitable for comparison or adaptation to their setting. Furthermore, the experimental comparisons with recent state-of-the-art PU learning methods that I previously requested have not been provided. Therefore, I will maintain my original score.

Reviewer_TVo2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PUCal, a simple and computationally efficient approach to biased positive-unlabeled (PU) learning that avoids explicitly estimating the instance-dependent propensity score. The method first trains a non-traditional classifier (NTC) to predict the labeling probability and then applies post-hoc probability calibration using a small fully labeled dataset. The key theoretical insight is that, under the Function of Posterior (FOP) assumption and a monotonicity condition, the true class posterior can be recovered through a one-dimensional transformation of the NTC output. The authors provide estimation error bounds and evaluate several calibration techniques. Experiments on image and medical datasets with six simulated labeling mechanisms show that PUCal, particularly with Beta Calibration or Isotonic Regression, generally achieves competitive or superior predictive performance and calibration compared with existing biased PU methods, while substantially reducing computational costs.

Critical strengths
  1. Simple and computationally efficient approach to biased PU learning. The main strength is the observation that explicit propensity score estimation can be replaced by one-dimensional post-hoc calibration under suitable assumptions. This avoids the computational complexity and potential optimization difficulties associated with jointly estimating the classifier and labeling mechanism.

  2. Meaningful empirical evaluation across different labeling mechanisms. The paper evaluates multiple calibration strategies and compares PUCal against established biased PU methods, including SAREM, LBE, and JERM.

Critical weaknesses
  1. The theoretical analysis contains potential issues that affect the claimed estimation guarantees. In particular, Remark 1 states that the Rademacher complexity of a parametric calibration family is O(1/n_cal). However, finite-dimensional parametric function classes generally exhibit Rademacher complexity of order O(1/sqrt(n_cal)) without further assumptions. Moreover, the proof of Theorem 4 appears to apply a Rademacher-based generalization bound to the cross-entropy risk without specifying sufficient boundedness or regularity conditions for the loss. These concerns are consequential because theoretical estimation guarantees constitute a central contribution of the paper. The authors should clarify or correct these results.

  2. Limited experimental evidence for the most challenging practical settings. Although the paper motivates PUCal partly by the difficulty of learning when positive-labeling frequency is low, all reported experiments use a labeling frequency of c = 0.5. It remains unclear whether the favorable performance and computational tradeoffs persist when substantially fewer positives are labeled.

  3. The experiments generate biased PU data artificially from fully labeled datasets. While S5 and S6 examine violations of FOP, these experiments do not fully establish robustness to realistic selection mechanisms, particularly when labeling depends strongly on factors unrelated to class membership. This limits the practical generalizability of the conclusions.

Questions for authors
  1. For the real-world benchmark datasets, how is the true posterior y(x), required to construct labeling strategies S2–S4, obtained? Is it approximated using the ORACLE classifier? If so, how does this affect the claim that these strategies satisfy the FOP assumption?

  2. How does PUCal perform under substantially lower labeling frequencies, such as c = 0.1 or c = 0.2?

1 comment

Critical Weaknesses 1

We stated in the answer to the Reviewer USUp that the due to the misprint the rate of convergence for parametric families was not stated correctly. However, we also explain that the rate of convergence in excess risk is conjectured to be O(1/n)O(1/n) in this case. Concerning Theorem 4 we are convinced that the proof provided in the appendix is correct. Obviously it concerns estimator y^(x)\hat{y}(x) obtained as calibrated estimator s^(x)\hat s(x) when both calibration function h^(x)\hat h(x) and estimator $\hat s(x) $ have been obtained as minimisers of empirical risk minimisation for log-loss only.

Q: For the real-world benchmark datasets, how is the true posterior y(x), required to construct labeling strategies S2–S4, obtained? Is it approximated using the ORACLE classifier? If so, how does this affect the claim that these strategies satisfy the FOP assumption?

In the case of real-world data, y(x)y(x) remains unknown and therefore must be estimated. Indeed, we estimate y(x)y(x) using the ORACLE method, which inevitably introduces some estimation error due to the limited amount of data or model misspecification. We can therefore assume that the FOP assumption is satisfied only approximately. However, it is worth noting that any potential deviation from the FOP assumption only strengthens our conclusions, as it demonstrates the effectiveness of the method in a setting where the assumption does not hold perfectly. Additionally, it is worth emphasizing that similar labeling strategies are used in other works on biased PU learning, e.g. in [1].

[1] Gong, C. and Wang, Q. and Liu, T. and Han, B. and You, J. and Yang, J. and Tao, D., Instance-Dependent Positive and Unlabeled Learning with Labeling Bias Estimation, IEEE Trans Pattern Anal Mach Intell (2021).

Q: How does PUCal perform under substantially lower labeling frequencies, such as c = 0.1 or c = 0.2?

We conducted additional experiments using a smaller value of c. The results, illustrating how the considered metrics (BACC, AE, and ECE) depend on computation time, for c=0.2c=0.2 and c=0.1c=0.1 are presented in the newly added Figures 4 and 5. As expected, decreasing the value of c leads to a deterioration in the performance of all methods, as a larger number of positive observations remain unlabeled, making the problem more challenging. For example, for the MIMIC-K dataset, the Balanced Accuracy averaged over labeling schemes and simulations for the two best-performing methods (PUCal-B and SAREM-B) is approximately 0.80 for c=0.5c=0.5 and 0.75 for c=0.2c=0.2. A similar effect can be observed for AE and ECE: decreasing cc leads to an increase in errors. Importantly, however, the overall picture remains unchanged: the proposed PUCal method achieves performance comparable to or better than that of methods requiring propensity score estimation, while being considerably faster.

Reviewer_USUpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on reducing the cost of estimating the propensity score, which is the key step in current biased positive-unlabeled learning methods, and proposes PUCal. PUCal calibrates a non-traditional classifier using a small fully labeled dataset, recovering class probabilities under an invertibility condition on the labeling mechanism, rather than explicitly modeling propensity score model. Experiments show competitive predictive performance and substantially lower runtime than propensity-based methods.

Critical strengths
  • The approach is simple, computationally efficient, and readily implemented using standard calibration methods.
  • Theorems provide an interpretable connection between score calibration and posterior recovery.
  • Including calibrated versions of competing PU methods improves comparison fairness.
Critical weaknesses
  • Assumptions are understated. Theorem 1 requires both FOP and strict monotonicity of w(t)=tg(t)w(t)=tg(t) (p. 4). FOP alone is insufficient, and the additional condition effectively restores order preservation. Claims framed solely around FOP should be qualified.
  • Novelty is insufficiently established. The distinction from prior threshold calibration is not adequately evaluated. Since monotone calibration changes classification through threshold adjustment, an NTC baseline with its threshold optimized on the same calibration set is essential (p. 7).
  • The rate claims are problematic. Remark 1 states that parametric classes have Rademacher complexity O(ncal−1)O(n_{\rm cal}^{-1}), which is generally incorrect. Its conjectured O(ncal−1)O(n_{\rm cal}^{-1}) absolute posterior error also appears to conflate probability estimation error with excess risk (p. 6).
  • Experimental coverage is limited. The experiments fix the labeling frequency at c=0.5c=0.5 (p. 8), leaving performance under scarce positive labels unclear. The representativeness and class composition required of the small calibration set also need clarification.
Questions for authors
  1. What thresholds are used to compute BACC? With positive temperature and a fixed threshold of 0.50.5, temperature scaling preserves NTC predictions. Why does PUCal-T’s BACC change with calibration-set size in Figure 2?
  2. How does PUCal compare with prior threshold calibration and NTC threshold tuning under the same labeled-data budget?
  3. Can the authors correct the rate claims and distinguish assumptions needed for posterior identifiability from those needed for estimation guarantees?
  4. Does the reported advantage persist at lower labeling frequencies and with highly imbalanced calibration sets?
2 comments

Q: What thresholds are used to compute BACC? With positive temperature and a fixed threshold of (0.5), temperature scaling preserves NTC predictions. Why does PUCal-T’s BACC change with calibration-set size in Figure 2?

For Balanced Accuracy, we used a threshold equal to π=P(Y=1)\pi=P(Y=1), which was estimated as the average of the estimated posterior probabilities. This is the optimal choice of threshold for Balanced Accuracy (cf [1]), in contrast to standard Accuracy, for which the optimal threshold is a fixed value of 0.5. Indeed, the value of the temperature parameter T does not affect Accuracy because the classification rule is logit(s(x))/T>0{\rm logit}(s(x))/T>0. However, importantly, the value of the parameter T affects Balanced Accuracy because, in this case, the classification rule takes the form logit(s(x))/T>σ−1(π){\rm logit}(s(x))/T>\sigma^{-1}(\pi). The size of the calibration set, in turn, affects the quality of the estimation of TT. Therefore, we observe changes in Balanced Accuracy depending on the size of the calibration set. An analysis of analogous plots for standard Accuracy shows that the value of TT, and consequently the size of the calibration set, indeed has no effect on Accuracy of PUCal-T.

[1] A. Menon, H. Narasimhan, S. Agarwal, and S. Chawla. On the statistical consistency of algorithms for binary classification under class imbalance. In Proceedings of the 30th International Conference on Machine Learning, ICML’ 13, pages 603–611, 2013.

Q: How does PUCal compare with prior threshold calibration and NTC threshold tuning under the same labeled-data budget?

Threshold calibration for NTC is a possible approach, but it is specifically intended for situations in which our sole objective is to obtain an optimal classification rule. In this work, we focus on the more challenging task of estimating posterior probabilities. The tables below show the performance of NTC and NTC (tuned), where the optimal threshold for Balanced Accuracy is selected based on the calibration set. As expected, NTC (tuned) performs significantly better than NTC and is generally comparable to PUCal in terms of Balanced Accuracy. However, NTC (tuned) performs considerably worse than PUCal in terms of ECE and AE because it does not correctly estimate posterior probabilities (it uses the same probabilities as NTC and only modifies the classification threshold). We also note that Theorem 1 is actually more general than it is stated, namely it holds true for all injective transformations hh. If hh is injective but not monotone then orderings according to s(x)s(x) and y(x)y(x) are not concordant and then NTC with tuned threshold would be inconsistent estimator of Bayes classifier. General version of Theorem 1 is now included.

\begin{array}{c|c|c|c} \text{Strategy} & \text{NTC} & \text{NTC (tuned)} & \text{PUCal-B} \\ \hline S1 & 0.576 & 0.665 & 0.706 \\ S2 & 0.594 & 0.697 & 0.699 \\ S3 & 0.558 & 0.720 & 0.681 \end{array} Table 1. Balanced accuracy of NTC, NTC (tuned), and PUCal-B under different strategies on the MIMIC-K dataset.

\begin{array}{c|c|c|c} \text{Strategy} & \text{NTC} & \text{NTC (tuned)} & \text{PUCal-B} \\ \hline S1 & 0.168 & 0.168 & 0.029 \\ S2 & 0.162 & 0.162 & 0.032 \\ S3 & 0.178 & 0.178 & 0.031 \end{array}

Table 2. Expected Calibration Error (ECE) for NTC, NTC (tuned), and PUCal-B on the MIMIC-K dataset.

\begin{array}{c|c|c|c} \text{Strategy} & \text{NTC} & \text{NTC (tuned)} & \text{PUCal-B} \\ \hline S1 & 0.143 & 0.143 & 0.087 \\ S2 & 0.141 & 0.141 & 0.066 \\ S3 & 0.160 & 0.160 & 0.086 \end{array}

Table 3. Approximation Error (AE) for NTC, NTC (tuned), and PUCal-B on the MIMIC-K dataset.

Q: Can the authors correct the rate claims and distinguish assumptions needed for posterior identifiability from those needed for estimation guarantees?

There is a misprint in Remark 1, for which we apologise: the Rademacher complexity for parametric families in the general case is obviously O(ncal−1/2)O(n_{\mathrm{cal}}^{-1/2}), not O(ncal−1)O(n_{\mathrm{cal}}^{-1}).

However, the point of the remark was, firstly, that the rate for parametric family is the same as for function classes of polynomial discrimination and, secondly, that in the case of parametric families bounding of excess risk in terms of Rademacher complexities is suboptimal, as excess risk should be equivalent to quadratic form related to Hessian H(θ∗)H(\theta^{*}), which with (θ^−θ∗)=Op(1/n1/2)(\hat\theta- \theta^*)=O_p(1/n^{1/2}) and boundedness of Hessian yields parametric Op(1/n)O_p(1/n) order of convergence.

The listed assumptions of Theorem 3 are used to obtain estimation guarantees as stated in the result. The identifiability of the posterior is ensured by existence of transform hh which implies that the posterior y(x)y(x) is uniquely determined by observable posterior s(x)s(x). As it follows form Theorem 3, assumption h∈Hh\in {\cal H} is not required there and h~∈H\tilde h\in {\cal H} is needed for estimation guarantees only.

Q: Does the reported advantage persist at lower labeling frequencies and with highly imbalanced calibration sets?

We conducted additional experiments using a smaller value of c. The results, illustrating how the considered metrics (BACC, AE, and ECE) depend on computation time, for c=0.2c=0.2 and c=0.1c=0.1 are presented in the newly added Figures 4 and 5. As expected, decreasing the value of c leads to a deterioration in the performance of all methods, as a larger number of positive observations remain unlabeled, making the problem more challenging. For example, for the MIMIC-K dataset, the Balanced Accuracy averaged over labeling schemes and simulations for the two best-performing methods (PUCal-B and SAREM-B) is approximately 0.800.80 for c=0.5c=0.5 and 0.750.75 for c=0.2c=0.2. A similar effect can be observed for AE and ECE: decreasing cc leads to an increase in errors. Importantly, however, the overall picture remains unchanged: the proposed PUCal method achieves performance comparable to or better than that of methods requiring propensity score estimation, while being considerably faster.

Combining a very high degree of class imbalance with a low labeling frequency results in an extremely challenging classification problem, as only a very small number of observations are labeled in such a setting. Therefore, we restricted our analysis to moderately imbalanced MIMIC datasets, in which the disease prevalence is approximately 20%.

Comments on the paper

2 comments

We thank all Reviewers for constructive reviews. We appreciate a recognition that the problem is novel and that proposed solution avoids estimation of the propensity score function. We address all concerns in separate rebuttals for every Reviewer. Most of raised concerns will be accounted for in revised version of the final manuscript. In particular, we conducted additional experiments, for example, to demonstrate the results under a lower labeling frequency and to evaluate an approach based on selecting the optimal threshold using a calibration set.

Area_Chair_i8VDOpenReview ↗

Reminder: Please Review the Author Responses

Dear Reviewer USUp and TVo2,

As the discussion period is coming to an end, please take a moment to review the authors' responses and update your ratings if your assessment has changed. If you have any remaining concerns, please feel free to share them in the discussion.

Thank you for your efforts!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.