- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies learning under selective labels, where labels are missing non-randomly, causing standard empirical risk minimization (ERM) to underrepresent certain target populations. To address this issue, the authors propose Adaptive Population-Risk Reconstruction (APR). The method combines label-selection probabilities with similarity to the unlabeled data distribution to assess the relevance of labeled samples to the target population. These relevance scores are then used to weight labeled samples and estimate risks for the Core and Target populations separately. Finally, a global reliability weight and a local gating mechanism adaptively combine a standard predictor with a risk-reconstruction predictor. The paper also analyzes sufficient conditions and limitations of risk reconstruction. Experiments on synthetic and real-data-based semi-synthetic datasets show competitive performance against the compared baselines.
Critical strengths
- The paper addresses a practically relevant problem, focusing on the distribution mismatch between labeled training data and the target population caused by selective labeling.
- Leveraging a small number of labeled samples relevant to the target population for risk reconstruction provides a meaningful approach to mitigating selective-label bias.
- The theoretical analysis examines conditions under which risk reconstruction is valid and cases where identification is impossible, providing insights into the method's applicability.
Critical weaknesses
-
Although the proposed Bridge identification mechanism has some novelty, the framework introduces several additional components, including complex sample weighting, global reliability estimation, and local gating. These components largely build on existing ideas in sample weighting and model ensembling, making the overall framework relatively complex while offering limited additional algorithmic novelty.
-
Although Section 6 provides component ablations and a comparison with Fixed Bridge Reweighting, more direct simplified baselines are still missing, such as using the risk-reconstruction predictor alone without prediction fusion. This comparison would help distinguish the contribution of risk reconstruction from that of the additional fusion mechanism and better justify the complexity of the overall framework.
-
The paper analyzes conditions for Bridge identification and consistency of the reconstructed risk, which has theoretical value. However, the consistency result relies on the estimated weights approximating reference weights that satisfy the risk-identification condition. It remains unclear under what conditions the proposed relevance scores and power-based weighting scheme satisfy this requirement. Therefore, the connection between the theoretical analysis and the practical method needs further clarification.
Questions for authors
Could the authors address the above weaknesses?
9 comments
Response to Weakness 2
We thank the referee for raising this point on adding baselines. We consider four ablations to assess the contribution of APR's key components: APR-SimpleWeight replaces the population-risk reconstruction weights with simple relevance-based weights ; APR-NoGlobal removes adaptive global reliability estimation by fixing the global mixing coefficient at ; APR-NoLocal replaces the observation-specific local gate with a constant estimated from the development set; and APR-NoFusion removes the final prediction fusion and uses only the reconstruction predictor . All other components are retained whenever applicable. The table below reports the performance differences between Full APR and its ablated variants under Valid Bridge Stress, where positive values favor Full APR.
The role of prediction fusion. An interesting but interpretable result is that APR-NoFusion achieves a slightly higher AUC on Test A (). This is consistent with the design of APR: Test A consists exclusively of Target observations, representing an extreme evaluation setting in which the reconstructed predictor is particularly well aligned with the test population. Consequently, incorporating the ordinary ERM predictor through fusion may slightly dilute the Target-specific ranking advantage of the reconstructed predictor. However, this does not imply that removing fusion improves overall predictive quality. In fact, Full APR achieves a lower Log-Loss on Test A (). This distinction arises because AUC measures ranking performance, whereas Log-Loss evaluates the quality of predicted probabilities and penalizes confident incorrect predictions. Fusion can therefore improve probability predictions even when it slightly reduces Target-specific ranking performance.
Importantly, Test A represents an extreme, Target-only evaluation setting, whereas Test B evaluates the mixed Core--Target population and is more representative of the population-level learning objective considered in this study. On Test B, Full APR outperforms APR-NoFusion in both AUC () and Log-Loss (). These results highlight the practical motivation for prediction fusion: it serves as a protective mechanism that moderates reliance on the reconstructed predictor by retaining a contribution from ordinary ERM. This design aims to reduce vulnerability to imperfect bridge reconstruction and excessive Target-specific adaptation, rather than maximizing performance on a purely Target population. The ablation results support its practical value for population-level prediction, while also revealing a modest trade-off in Target-only AUC.
The contributions of the remaining components. Under the tested Valid Bridge Stress setting, these three ablations show measurable performance benefits from the full weight construction, global reliability adjustment, and local adaptation relative to their specified alternatives. Replacing the proposed reconstruction weights with simplified weights (APR-SimpleWeight) reduces performance across both test settings, with AUC differences of on Test A and on Test B. Removing global reliability adjustment (APR-NoGlobal) also consistently deteriorates performance, confirming the usefulness of accounting for the overall reliability of reconstructed Target evidence. Similarly, removing local adaptation (APR-NoLocal) produces substantial performance degradation, particularly in AUC ( on Test A and on Test B), supporting the importance of observation-specific adaptation. All three ablations also exhibit higher Log-Loss than Full APR, with positive performance differences and 95% confidence intervals excluding zero across both test settings.
Overall, these results demonstrate that the proposed weight construction, global reliability adjustment, and local adaptation each provide measurable contributions to APR's predictive performance. Prediction fusion serves a complementary role, improving probability prediction and mixed-population performance while allowing a small reduction in Target-only ranking accuracy. Together, the ablations provide empirical justification for the distinct components of the full APR framework.
| Ablation | AUC — Test A | AUC — Test B | Log-Loss — Test A | Log-Loss — Test B |
|---|---|---|---|---|
| APR-SimpleWeight | +0.0279 [+0.0206, +0.0351] | +0.0105 [+0.0056, +0.0154] | +0.0134 [+0.0089, +0.0179] | +0.0056 [+0.0034, +0.0078] |
| APR-NoGlobal | +0.0103 [+0.0088, +0.0119] | +0.0033 [+0.0021, +0.0045] | +0.0054 [+0.0041, +0.0068] | +0.0021 [+0.0014, +0.0028] |
| APR-NoLocal | +0.0278 [+0.0240, +0.0316] | +0.0115 [+0.0086, +0.0144] | +0.0133 [+0.0091, +0.0176] | +0.0062 [+0.0042, +0.0082] |
| APR-NoFusion | -0.0053 [-0.0060, -0.0046] | +0.0057 [+0.0041, +0.0073] | +0.0089 [+0.0048, +0.0130] | +0.0065 [+0.0046, +0.0084] |
Response to Weakness 3 (Part 1/3)
We thank the reviewer for this helpful comment. To clarify the connection between theory and implementation, we have developed additional theoretical results for APR's actual relevance-score transformation and power-based weighting scheme, without changing the algorithm. We summarize the main conditions, results, and proof arguments below. Detailed statements, technical assumptions, proofs, and illustrative constructions have been incorporated into the new revision of paper (Appendix B).
1. Conditions for valid relevance scores.
Let , , and . APR constructs the population counterpart of its relevance score as
We establish a score-validity lemma with two implications. First, if the selection-surprise and affinity signals for Target observations multivariate first-order stochastically dominate those for Core observations conditional on , then
for thresholds with positive tail probability.
Second, suppose a measurable Target-like region satisfies
where and . If normalization and clipping preserve a strict separation between the transformed signals, then
Proof sketch. The product of the nonnegative bridge signals is coordinatewise nondecreasing, as are APR's robust normalization, prevalence adjustment, and sharpening. Stochastic dominance therefore preserves Target enrichment, while stronger signal separation yields a strict score gap when endpoint clipping is nondegenerate.
These conditions explain when APR's implemented score is informative. Importantly, is a relevance score, not a calibrated posterior probability of latent Target membership.
Response to Weakness 1
We thank the reviewer for this helpful comment. We agree that several components of APR build on existing ideas in sample weighting and model ensembling. However, the main novelty of APR lies in using bridge observations to reconstruct population risk from selectively observed labels. Rather than simply correcting selection bias through sample weights, APR identifies labeled observations that may contain useful information about the under-observed Target population and uses them to reconstruct the Core and Target risk components. Importantly, bridge discovery relies on observed covariates and label-availability information, without requiring known Core/Target identities, additional outcome labels, or a fully specified outcome-dependent selection mechanism. This makes it practical in settings where collecting additional Target labels is difficult or costly. The remaining components serve distinct purposes. Global reliability controls how much we can trust the reconstructed information overall, while local gating determines where it should be used. These two mechanisms are complementary: one controls the overall strength of adaptation, and the other controls its use for individual predictions.
To test whether these components are useful, we added four ablation variants: APR-SimpleWeight, APR-NoGlobal, APR-NoLocal, and APR-NoFusion. The results show clear benefits from the full design in the tested Valid Bridge Stress setting. Compared with APR-SimpleWeight, full APR improves AUC by 0.0279 on Test A and 0.0105 on Test B. Removing global reliability also reduces performance. Likewise, full APR improves AUC over APR-NoLocal by 0.0278 on Test A and 0.0115 on Test B. The APR-NoFusion results reveal an interesting trade-off. On Test A, which contains only Target observations, removing fusion slightly improves AUC by 0.0053. This is expected because the reconstructed predictor is especially suited to Target-only evaluation. However, full APR achieves better Log-Loss on Test A (+0.0089 improvement). More importantly, on Test B, which represents the mixed deployment population, full APR improves both AUC (+0.0057) and Log-Loss (+0.0065). The results are consistent with the intended stabilizing role of fusion. Relative to NoFusion, full APR improves mixed-population AUC and Log-Loss and Target-only Log-Loss, while reducing Target-only AUC. This ablation establishes a performance trade-off in the tested setting, rather than directly isolating robustness to reconstruction error.