- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of selecting negative samples from unlabeled data in pairwise PU learning by proposing the PEPU sampling framework. PEPU introduces an external model as a prior, which evaluates the prior credibility of candidate samples as negative samples. The pairwise utility is measured by the BPR loss of the current PU model. Multiplying these two values yields the sampling distribution, thus selecting negative samples that are both credible and valuable during training. Experiments were conducted on two synthetic PU settings, CIFAR-PU, and three recommendation datasets, with PEPU achieving the best results among all compared methods.
Critical strengths
- The motivation behind the problem is clear. The difficult samples in U could be true negative samples or hidden positives. Using external priors to assess credibility while preserving the model's difficulty is a natural and practical approach. The entire paper revolves around "difficulty" and "credibility," with clear logic.
- The analytical experiments support the method. OVL auditing, prior degradation, and frontier experiments explain the inversion of Prior-only and Hard-only, and show that PEPU smoothly degenerates to Hard-only when the prior fails.
Critical weaknesses
- The innovation is limited. This paper combines "selecting difficult negative samples" (e.g., Robinson et al. 2021) with "using external information to determine the reliability of negative samples" (e.g., KGPolicy), but both approaches have already been studied. The authors need to explain why this combination is better than other combinations. The theoretical part also does not fully explain why this method allows the model to learn better.
- The experiments are insufficient to support the claim of a unified framework. The paper claims that PEPU is a general framework that can use various external priors, but each setting uses only one fixed prior and construction method, and only one training method, BPR. It lacks ablation studies of prior sources, construction methods, and downstream PU methods. Furthermore, the PU experiments mainly rely on data created by the authors, with improvements relative to the strongest ablation only ranging from 0.001 to 0.01 with no standard deviations reported.
- No comparison with the most relevant methods. The paper mentions several methods most similar to PEPU in concept, but none of them were compared in the experiments: methods that use external information to pick negative samples (such as KGPolicy), methods for sampling hard negative samples (such as DENS), and two-step methods for picking reliable negative samples within the PU (such as Spy). These are the most suitable competitors for evaluating PEPU. Additionally, if feasible, it is recommended to also compare it with PU methods such as nnPU. If comparisons are not possible due to differences in class priors or labeling settings, please explain the reasons.
Questions for authors
- When there is systematic prior bias, will PEPU be worse than Hard-only? How can harmful priors be identified in practice?
- According to Appendix Table 4, the contamination (hidden positive) rate of PEPU is higher than that of Prior-only (0.18 vs 0.10). So where does the benefit come from? Please compare the contamination rate and difficulty (BPR loss) of sampled negatives from prior-only, hard-only, and PEPU.
- On CIFAR-PU, the OVL of the DINOv2 prior is only 0.02 (Fig. 1e), indicating that it can already distinguish hidden positive samples and true negative samples very well. Under the same PU data partitioning, if DINOv2 is frozen and only one classification head is trained, what is the test AUC? This comparison helps to determine what additional benefits PEPU can bring compared to the simple approach of directly utilizing external model features.
I would be willing to raise my score if the suggested experiments are conducted.
2 comments
I thank the authors for their thoughtful responses, which have partially addressed my concerns. However, the experimental results I previously requested, particularly comparisons with relevant methods and the frozen-DINOv2 baseline, have not yet been provided. Therefore, I will maintain my original score.
We would like to thank the reviewer for the thoughtful and constructive feedback. In the following, we address the questions and concerns raised in the review.
Methodological contribution and related comparisons (Weaknesses 1 and 3): The cited methods address different aspects of negative selection. Robinson et al. (2021) use similarity to identify hard negatives; KGPolicy learns a knowledge-graph-guided sampling policy; DENS uses disentangled item factors to guide recommendation sampling; Spy identifies reliable negatives through two-stage PU selection; and nnPU addresses pointwise PU risk estimation. Although related in motivation, these methods differ from PEPU in their information sources, sampling mechanisms, or learning objectives. PEPU investigates how independently derived external credibility information can guide negative sampling while retaining model-dependent pairwise utility. Specifically, external information defines a fixed credibility reference, while the current BPR loss determines sampling utility. These signals are combined through an explicit reference-deviation constraint, rather than simply combining two sampling heuristics. Our comparisons include RNS, DNS, AOBPR, AHNS, and SRNS, together with Prior-only and Hard-only, under matched BPR protocols. The theoretical contribution also goes beyond exponential reweighting itself. Proposition 3.4 derives the optimal sampler, Theorem 3.5 establishes the complete utility–reference-deviation Pareto frontier, and Theorem 3.6 establishes optimization consistency with the ordinary BPR gradient.
Unified framework and experimental scope (Weakness 2 and Question 3): PEPU's unification lies in its common reference-based sampling formulation, instantiated with task-specific external information: an auxiliary view in controlled PU learning, frozen DINOv2 representations in image classification, and LLM-derived item representations in recommendation. BPR is the intended scope of this work. Pointwise PU methods such as nnPU and contrastive objectives such as InfoNCE address different learning objectives. Our evaluation covers controlled PU benchmarks constructed by hiding positive labels, real CIFAR-10 images under a brightness-dependent SAR mechanism, and real-world implicit-feedback datasets (Amazon-book, Yelp, and Steam). PEPU achieves the best AUC across all six settings, improving over the strongest ablation by 0.0013–0.0106. Appendix Table 4 reports three-seed results for the controlled PU budget sweep; at the default budget, the standard deviations are 0.0002 and 0.0014 in the two settings. DINOv2 is used exclusively to construct the external prior, while a separately initialized ResNet-18 learns from the original images using BPR, without shared parameters or gradients. A frozen-DINOv2 classifier evaluates direct feature transfer rather than prior-guided sampling. Our matched Prior-only and PEPU comparison uses the same DINOv2 prior and downstream model, isolating the contribution of model-dependent utility. Moreover, OVL = 0.02 measures prior discrimination, not the test AUC of a separately trained classification head. We do not claim superiority over direct DINOv2 feature transfer.
Robustness to systematically misleading priors (Question 1): Section 4.2 shows that PEPU approaches Hard-only as the prior loses its discriminative information through noise injection. A systematically reversed prior can misguide the reference and potentially make PEPU perform worse than Hard-only. In practice, prior reliability can be assessed using independent domain knowledge or a trusted validation set when available. Our framework assumes meaningful relative credibility guidance and does not claim automatic correction of arbitrary prior bias.
Sampling quality and utility–contamination trade-off (Question 2): Table 7 reports expected BPR utility and hidden-positive contamination along the Prior-only-to-PEPU sampling path. At the default budget, utility increases from 0.113 to 0.234, while contamination increases from 0.098 to 0.182. Table 4 reports corresponding training results, with AUC improving from 0.8765 to 0.8840 and contamination increasing from 0.098 to 0.179. Table 1 further compares downstream AUC across Prior-only, Hard-only, and PEPU, although it does not provide a matched three-way contamination and utility comparison. These results illustrate the central mechanism of PEPU: model-dependent utility favors more informative pairs, while the deviation budget controls departure from the credibility reference. The objective is to balance negative credibility and pairwise utility, rather than independently minimize contamination.
We believe these clarifications help distinguish PEPU from related sampling approaches and clarify the scope of the proposed framework and its empirical validation.