Power-Optimal Risk Control via Standardized Margin Calibration

Machine learning can screen far more candidates than can be experimentally validated, creating a need for selection procedures that identify promising candidates while controlling risk before costly experimental validation. We propose Uncertainty-Normalized Empirical Calibration (UNEC), a $p$-value-free framework that ranks candidates by uncertainty-normalized predictive margins and directly calibrates a selection cutoff using held-out losses. UNEC supports flexible prediction models, general bounded losses, and diverse selection targets. We study marginal discovery risk (MDR) and selective discovery risk (SDR), with SDR reducing to the false discovery rate under binary loss. We establish finite-sample MDR control for an adjusted rule and asymptotic SDR control with explicit limiting risk and power. Under a location–scale model with a common one-sided binary target, we further show that UNEC asymptotically attains the oracle power benchmark among observation-wise selection rules. Simulations and molecular screening applications in solubility, toxicity, and lipophilicity demonstrate favorable risk–power trade-offs and substantial computational gains across binary and graded-loss settings.