- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper deals with biased selection PU model under case control scenario, when additional information is given whether the item will undergo possible labeling or not. Namely, labeling proces itself is proceded by verification, only observations which get positive verification status (A=1) are potentially labeled under SCAR i.e. positive observations are labeled with constant probability c, and the remaining verified positives and all verified negatives are unlabeled. The observer is given a sample L of labeled observations, sample U of observations without class indicators pertaining to general population and a sample V of observations without class indicators but with corresponding information whether particular item was verified or not. Thus, the fact that labeled observations do not follow positive class distribution, as in SCAR case, is due to the fact that verification depends on predictors. The aim is to generate new observations from positive class by adapting diffusion process approach using available data. This is based on estimation of density ratio of densities of positively verified items to density of labeled items and the main theoretical result (Theorem 2) establishes bound on total variation distance between positive class distribution and -weighted distribution of general population. Experiments show usefulness of the approach in generating items from the positive class over methods which disregard inherent bias of labeled observations.
Critical strengths
The approach nicely disentangles process which leads to biasedness of labeled observations into verification process and labeling process itself. By using conditional independence of verification and class indicatator given predictors () (which is not true when A is replaced by labeling indicator S), this allows for estimation of density ratio and, consequently, distribution of positive class.
Critical weaknesses
The main weakness is that under conditional independence assumption and with observability of V sample, the problem becomes conceptually rather simple. Namely, in the present setting propensity score is proportional to posterior probability of being positively verified, and as V sample is available, is estimable, while estimation of the proportionality constant can be avoided by suitable normalization. Moreover, crucial ratio can be estimated via estimation of . The second problem is that in the present format the paper is very hard to read, with essential information mostly contained in the appendix. In particular, the algorithms for the method are not given in the main body of the paper, but only on p. 21 in the appendix, and the essence of Inverse Probability Weighting ( IW) and Selected Reference (SW) approach, which is crucial, is also given there (appendix B2). In my opinion the paper requires substantial restructuring.
Questions for authors
- Estimation of can be performed using many alternative methods such as direct density-ratio estimation methods e.g. relative unconstrained least-squares importance fitting (Yamada et al. 2012). Why the method employed in the paper is preferable ?
3 comments
The Authors have improved the readability of the manuscript in its revised version, which is now much easier to follow. But I believe alternatives to estimation of density ratio by minimising the balanced risk pertaining to log-loss should be explored more thoroughly. Indeed, the method applied is essentially classification based approach to estimate the ratio, and this is known to work poorly for high dimensions and/or well separated densities. Still, I am inclined to give the Authors the benefit of doubt. I retain my score.
Thank you for recognizing the improved readability and for pressing us on density-ratio estimation. We agree that our earlier 200-update neural relative-loss control was not a sufficient comparison. New Appendix G, referenced from Sections 4.2 and 4.3, compares kernel uLSIF and RuLSIF with logistic estimators on CelebA64, first for weighted targets and then for generated images.
Setup. All estimators use the same 512-dimensional frozen CLIP features, five source blocks and 100,000 images in U: linear logistic, Gaussian-kernel logistic (sharing the least-squares basis, to separate loss from basis), uLSIF, and RuLSIF with ρ = 0.1 and 0.5. Each selects hyperparameters by its own validation loss on three folds of the source fitting data; neither target attributes nor target TV enter any choice. After source validation favored grid boundaries, we expanded the search to 100–500 centers, bandwidth factors 0.25–4 and ridge values 1e-6 to 1.
Weighted targets. Joint Smiling×Male TV×100 of SR's continuous weights to the fixed positive reference (predicted attributes; lower is better), with ESS as % of U; means ± descriptive SE over five reused blocks (Table 38):
| Estimator | Target TV | ESS (%) |
|---|---|---|
| Previous linear logistic | 19.65 ± 0.38 | 69.71 ± 0.84 |
| Updated linear logistic | 8.53 ± 0.38 | 46.33 ± 0.77 |
| Kernel logistic | 15.65 ± 0.84 | 57.29 ± 1.60 |
| uLSIF | 19.89 ± 0.49 | 68.78 ± 0.68 |
| RuLSIF ρ=0.1, direct | 19.42 ± 0.76 | 68.71 ± 1.21 |
| RuLSIF ρ=0.1, inverse | 17.72 ± 0.75 | 65.49 ± 1.20 |
| RuLSIF ρ=0.5, direct | 23.22 ± 0.47 | 76.56 ± 0.90 |
| RuLSIF ρ=0.5, inverse | 11.72 ± 0.67 | 48.89 ± 2.54 |
First, the comparison exposed a consequential choice in our own estimator: all ten updated linear fits select ridge 0.1, missing from our previous grid, which lowers SR target TV by 11.12 ± 0.10 (paired SE) in every block. Second, the ranking depends on the reference. Held-out source AUC is 0.79–0.83 against all records but 0.68–0.73 against selected records. With the more separable all-record reference, every least-squares rule (18.04–23.63) beats updated linear and kernel logistic (24.64 and 24.52; Table 39), consistent with your point about well-separated sources, although the two references define different targets. Better source log loss alone does not give a better correction. Third, with SR's fixed source threshold, RuLSIF ρ=0.5 (4.18 ± 0.71) and updated logistic (4.32 ± 0.51) come close to, but do not improve on, the existing 3.80 ± 0.55.
Generated images. With the diffusion base frozen, we trained 70 new guidance potentials (width 96, 20,000 updates; 200-step sampling; 1,024 images per block) and reused the 25 existing controls. Kernel runs use the initial source grid, so their targets differ from the table above: uLSIF and RuLSIF ρ=0.1 with both references and all weight transformations. The linear follow-up was motivated by the target results. Paired changes in generated outcomes (mean ± paired SE over the same blocks; coverage at k=5 in pp; Tables 40–42):
| Comparison | Joint TV×100 | FID | Coverage |
|---|---|---|---|
| Updated − previous SR soft | −4.79 ± 0.45 | −0.24 ± 0.11 | +0.92 ± 0.20 |
| Updated SR soft − previous SR threshold | +3.36 ± 1.04 | +0.11 ± 0.14 | +0.25 ± 0.19 |
| RuLSIF inverse SR − previous SR threshold | +8.59 ± 1.22 | +0.34 ± 0.08 | −1.15 ± 0.38 |
The estimator improvement reaches generation: joint TV and coverage improve in all five blocks. The existing SR threshold still gives the lowest joint TV (18.52 ± 0.85); every kernel method is worse than it in every block, and refitting the thresholded linear estimate gives 18.96 ± 1.23. With both linear soft fits updated, SR improves FID by 2.93 ± 0.32 and coverage by 1.95 ± 0.38 pp over the all-record reference but raises joint TV by 2.50 ± 0.62. Target gains also outpace output gains: the updated soft target predicts 78.19% Smiling but generation reaches 64.84%, and its TV to its own target rises from 7.07 to 13.36 (Table 43).
Result. 204 of 1,050 kernel-logistic candidates had an unconverged fold, mainly at small ridge, and were excluded; all selected fits converge, and least-squares fits are direct linear solves. Some selected kernels sit at grid boundaries, so these results hold within the stated ranges. Within them no estimator wins across references and weight rules: on high-dimensional features, loss, regularization, reference and weight transformation need to be examined together.
Thank you again; this suggestion improved our own estimator in both targets and generated images.
Though the presentation in our initial submission was not really easy to follow, we are very grateful for your careful review and deep understanding of our method. As you point out, the cancellation is straightforward under our assumptions, but the main text made the method harder to follow than necessary. We have therefore reorganized the paper around the data flow and sharpened the statement of our contribution.
1. The method now reads from the main text. Sections 2–3 follow the data through identification, weight estimation and generation.
Detailed objectives and pseudocode are in Appendix C.1. Sections 4.1–4.4 then examine record information, weighting strategies, generation and assumption violations in turn.
2. What the simple identity buys. A generator can fit labeled positives perfectly and still reproduce the wrong proportions within the positive class. The identity says which reference fixes this. Under shared selection, constant positive reporting and overlap, rA=fl/fA, where fA includes both classes, and (rA/EPrA)P=P+. SR estimates these corrective weights without a propensity model.
The identity leaves three practical questions open, and the paper answers each. How does ratio error affect the target? Theorem 2 bounds the target error by the relative ratio error for any nonnegative estimator, and Section 3.3 separates it from the error of generating from the target. Do the records carry information beyond reference size? With equal reference counts, selected rather than random references raise weighted-resampling coverage by 2.89 ± 0.70 percentage points on MNIST under strong selection and by 1.86 ± 0.30 on biased CelebA, at a CelebA precision cost of 1.60 ± 0.49 (five paired repetitions; SE); constant selection shows no comparable gain (Section 4.1). Does the correction reach generated samples? On MNIST, SR lowers intensity-group TV at matched predicted purity; on CelebA it improves FID but worsens joint composition (Section 4.3).
As you note, normalized IPW also removes the unknown scale, and in our matched MNIST controls a correctly specified IPW model performs within one paired SE of SR (Section 4.2). SR gets there without specifying a selection model. Kato et al. (2023) used exposure records for PU classification, and conditional energy guidance is established. Our contribution is the distribution-recovery formulation for generation, its estimator and error analysis, and the controlled study of when the correction helps.
3. Why logistic estimation? Two reasons. It is a direct ratio estimator that fits naturally on image networks: with balanced source classes, the population-optimal logit is log(fl/fA), so exponentiation gives the ratio without estimating either density. Its excess population logistic loss also controls the target-distribution error when ratios are bounded and selection probabilities are bounded away from zero (Appendix B.2.1).
RuLSIF is a natural alternative and slots into the same pipeline: Theorem 2 holds for any nonnegative estimator, and Appendix B.2.3 shows how relative ratios map to the ordinary ratio, including when direct normalization preserves the target. In matched MNIST controls with the same network and a fixed 200-update budget, neural relative least-squares estimates have higher joint class-by-intensity TV than logistic SR in every tested setting, though one gives better within-positive composition in one setting (Appendix E.2.2). This is a neural adaptation rather than kernel RuLSIF with its own model selection, so it shows that logistic is a sound choice, not that it is the best one.