acceptodds
Under review as a conference paper at ICLR 2027

Power-Optimal Risk Control via Standardized Margin Calibration

Abstract

Machine learning can screen far more candidates than can be experimentally validated, creating a need for selection procedures that identify promising candidates while controlling risk before costly experimental validation. We propose Uncertainty-Normalized Empirical Calibration (UNEC), a -value-free framework that ranks candidates by uncertainty-normalized predictive margins and directly calibrates a selection cutoff using held-out losses. UNEC supports flexible prediction models, general bounded losses, and diverse selection targets. We study marginal discovery risk (MDR) and selective discovery risk (SDR), with SDR reducing to the false discovery rate under binary loss. We establish finite-sample MDR control for an adjusted rule and asymptotic SDR control with explicit limiting risk and power. Under a location–scale model with a common one-sided binary target, we further show that UNEC asymptotically attains the oracle power benchmark among observation-wise selection rules. Simulations and molecular screening applications in solubility, toxicity, and lipophilicity demonstrate favorable risk–power trade-offs and substantial computational gains across binary and graded-loss settings.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Risk-controlled selection, Uncertainty quantification, Empirical calibration, General loss functions, False discovery rate control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_A7jROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of candidate screening by proposing a p-value-free approach based on a score that combines the predicted value and its estimated variance. The score measures the normalized distance of each candidate from a decision boundary. A calibration procedure is then used to control selective discovery risk (SDR) and marginal discovery risk (MDR).

The theoretical contributions include finite-sample MDR control, asymptotic SDR control, and an analysis of statistical power. The authors further establish power optimality under a location-scale model. The proposed method is evaluated on diverse synthetic and real-world datasets.

Critical strengths
  1. Clear presentation and practical efficiency. The paper is well written, and the proposed method is simple to implement and computationally efficient. Its p-value-free formulation makes it an appealing alternative to conventional screening procedures.

  2. Theoretical analysis and empirical validation. The paper provides theoretical guarantees for MDR and SDR control, together with a statistical power analysis. The theoretical results are complemented by experiments on diverse synthetic and real-world datasets, supporting the practical relevance of the approach.

Critical weaknesses
  1. Limited novelty and generality of the theoretical results. The proposed calibration procedure appears closely related to existing conformal risk control methods, and the paper does not sufficiently clarify its methodological novelty relative to these approaches beyond the setting difference. Moreover, theory only establishes asymptotic SDR control. Furthermore, the power optimality result is established under a restrictive location-scale model that appears particularly suited to the proposed score. It is unclear how realistic this assumption is or whether the optimality result extends to more general settings.
  2. Limited empirical advantage in statistical power. In many experimental settings, the proposed UNEC method achieves lower power than the SCoRE baseline. This raises concerns about its practical benefits and whether the proposed approach offers a meaningful improvement over existing methods.
Questions for authors
  1. The calibration procedure appears similar to existing conformal risk control approaches. Could the authors clarify the key methodological differences and identify which aspects of the proposed framework are fundamentally new beyond the difference in the setting?

  2. The power optimality result relies on a specific location-scale model that appears tailored to the proposed score. How realistic is this model for the applications considered? To what extent does the optimality result remain valid under deviations from these assumptions?

  3. The paper considers three main baselines: CS, NP, and SCoRE. However, some experiments compare UNEC against CS and NP, while others compare it against SCoRE. Could the authors provide consistent comparisons against all three baselines across the experiments?

  4. Why is the benefit of UNEC more pronounced for small q values?

5 comments
AuthorsOpenReview ↗

Response to Reviewer A7jR (Part I)

Thank you very much for your feedback:

Q1. ..clarify the methodological differences and identify which aspects of the framework are new..?

The main contribution of UNEC is the combination of uncertainty-normalized ranking and direct empirical risk calibration, rather than the introduction of a different risk-control objective. As described in Section 3 (Algorithm 1), UNEC ranks candidates using the standardized predictive margin Z(x;c)=(μ^(x)−c)/σ^(x)Z(x;c)=(\widehat{\mu}(x)-c)/\widehat{\sigma}(x) and determines the selection cutoff directly from held-out losses, without constructing conformal pp- or ee-values.

Computationally, UNEC calibrates selection thresholds using sorted scores and cumulative losses, whereas the evaluated SCoRE implementations require repeated conformal ee-value calculations across candidate thresholds and test observations. As discussed in Section 5, these calculations dominate SCoRE’s selection cost, with computational complexity scaling approximately as m(n+m)m(n+m) in our implementation. This difference is reflected in Figure 2, where UNEC requires approximately 1.3 seconds per repetition, compared with 35–37 seconds for SCoRE over the full grid of qq.

Also, existing approaches differ in their constructions and guarantees. Conformal risk control (CRC; Angelopoulos et al., 2024) calibrates predictive procedures to control expected loss under a monotonicity condition, rather than directly controlling the average loss among selected observations. CS (Jin and Candès, 2023) combines conformal pp-values with multiple testing to control FDR, while SCoRE (Bai and Jin, 2026) uses risk-adjusted conformal ee-values for general selective risk control. The NP method (Qin et al., 2025) uses likelihood-ratio-based ranking with threshold calibration. In contrast, UNEC constructs its ranking from estimated conditional location and scale and directly calibrates empirical MDR or SDR.

A key strength of UNEC is its broad scope and flexibility. The framework supports general bounded losses beyond binary false-selection errors, allowing selection risk to reflect graded errors or application-specific costs. By adapting the standardized margin and corresponding loss, the same calibration procedure readily accommodates common one-sided targets, observation-specific thresholds, and two-sided interval targets, without requiring a separate testing procedure for each setting (Section 3 and Appendix A).


Q2. The power optimality result relies on a specific location-scale model..How realistic is this model for the applications considered? To what extent does the optimality result remain valid under deviations from these assumptions?

The location–scale assumption provides a sufficient condition for the power optimality of UNEC, rather than a restriction on its applicability or risk-control guarantees.

The location–scale family is substantially broader than the Gaussian model. It accommodates nonlinear conditional means, heteroscedasticity, and general continuous error distributions, provided the standardized error distribution is common across covariate values. Under this model, Section 4.2 establishes that

=1-F_\epsilon\left(-\frac{\mu(x)-c}{\sigma(x)}\right).$$ Hence, the oracle standardized margin preserves the ordering of the posterior success probability without requiring Gaussian errors or explicit estimation of the residual distribution. We acknowledge that requiring a common standardized error distribution is a restriction. Outside this family, conditional distributions may change shape across covariates, and their first two moments do not generally determine the posterior success probability. The standardized margin may then fail to produce the optimal ranking. Nevertheless, `UNEC` remains applicable because its risk calibration does not depend on this assumption. In particular, `Theorems 1 and 2` establish risk-control guarantees under the corresponding calibration, exchangeability, and regularity conditions, without requiring a location–scale model. Our empirical results further examine deviations from this structure. `DGPs 4–5` (Appendix C.2) are covariate-dependent Gaussian mixtures that fall outside the location–scale family, **yet `UNEC` retains competitive risk–power performance (`Figures 1–2`)**. Moreover, our new sensitivity analysis in `Appendix E (Figure 17)` evaluates nine scale-estimation specifications. In DGP 4, at $q=0.10$, uncertainty-normalized scores achieve `24.97%–38.84% power`, compared with `2.44%` for `UNEC-Raw`, while attaining empirical SDR closer to the nominal level. These findings support UNEC’s empirical usefulness beyond the location–scale family. Although UNEC delivers strong power in practice, these results do not establish universal power optimality under arbitrary conditional distributions. Extending the optimality result to such distributions remains an open theoretical question.
AuthorsOpenReview ↗

Response to Reviewer A7jR (Part II)

Q3. The paper considers three main baselines: CS, NP, and SCoRE... Could the authors provide consistent comparisons against all three baselines across the experiments?

CS and NP are designed for FDR control under binary false-selection loss and do not directly provide SDR control for general bounded losses. This is why we compare UNEC with CS and NP in Figure 1, where SDR reduces to FDR, and with SCoRE in Figure 2, where the experiments involve general bounded losses and require SDR control.

These comparisons are also connected. Corollary 5.4 of Bai and Jin (2026) establishes that a binary-loss specialization of SCoRE is equivalent to clipped CS under the same underlying score and appropriate parameter settings. Thus, our CS comparison also covers this particular specialization of SCoRE.

Overall, the choice of baselines reflects their respective risk-control capabilities rather than an omission of relevant comparisons.

Q4. Why is the benefit of UNEC more pronounced for small q values?

We attribute UNEC's observed power advantage at small qq to the combined effects of uncertainty-aware ranking and direct empirical risk calibration, rather than to a universal advantage over conformal methods.

First, UNEC accounts for local predictive uncertainty when ranking candidates, distinguishing candidates with similar predicted means. Theorem 3 (Section 4.2) justifies this ranking under the location-scale model. Empirically, Appendix C.3 (Figure 6) demonstrates favorable power gains at comparable empirical FDR levels, while the additional experiments in Appendix E (Figure 18) show that the improvement over UNEC-Raw persists across different uncertainty estimators, particularly at small qq.

Second, UNEC selects the score cutoff directly from the empirical loss curve (Algorithm 1), whereas CS and SCoRE rely on conformal pp-values or ee-values followed by BH or e-BH, whose rejection requirements may limit discoveries under stringent risk constraints. This direct calibration can permit less conservative selections, particularly at small qq, when the candidate ranking and empirical risk estimates are sufficiently informative.

Together, uncertainty-normalized ranking and direct empirical calibration help explain UNEC's observed power advantage at small qq. This advantage is empirically supported and task-dependent, as illustrated by the additional Merck experiments in Appendix D.3, rather than a universal guarantee.

W1. Limited empirical advantage in statistical power.

Neither UNEC nor SCoRE consistently outperforms the other in statistical power across tasks. However, UNEC achieves competitive power with substantially lower computational cost, making it a practical alternative for large-scale risk-controlled selection.

We added two molecular screening benchmarks, 3A4 and LOGD, from the Merck Molecular Activity Challenge (Appendix D.3, Figures 15–16). The results show task-dependent power differences: SCoRE-homo achieves higher power on 3A4 at comparable empirical SDR, whereas UNEC achieves higher power than all three SCoRE variants on LOGD while maintaining empirical SDR close to the nominal level. Notably, UNEC also requires substantially less computation on both benchmarks, as shown below.

3A4

qqUNECSCoRE-dtmSCoRE-homoSCoRE-hete
0.109.82 / 27.280.78 / 2.909.52 / 37.2010.73 / 25.70
0.2020.06 / 70.664.02 / 15.8620.06 / 79.0719.89 / 72.23
Runtime (s)15.11307.32307.32307.32

LOGD

qqUNECSCoRE-dtmSCoRE-homoSCoRE-hete
0.108.37 / 8.771.10 / 0.946.51 / 5.936.35 / 3.69
0.2019.53 / 30.284.22 / 5.7018.05 / 24.7017.76 / 17.33
Runtime (s)18.40266.38266.38266.37

Each risk-level cell reports mean empirical SDR / mean power, both in percent. Runtime is averaged per repetition over the full grid of qq, including model fitting, prediction, and selection.

W2. Lack of finite-sample SDR control.

We have addressed this limitation by developing a conservative, concentration-based variant of UNEC that achieves finite-sample SDR control for general bounded losses (Appendix B.4). This extends the theoretical guarantees of UNEC beyond the asymptotic SDR control established for the standard procedure (Theorem 1).

Specifically, the new variant constructs simultaneous upper confidence bounds for the conditional risk Q(z)=E[L∣Z≥z]Q(z)=\mathbb{E}[L\mid Z\geq z] using an order-statistic argument, a KL-based concentration inequality, and a union bound. With an adjusted risk level qα=(q−α)/(1−α)q_\alpha=(q-\alpha)/(1-\alpha) for 0<α<q0<\alpha<q and a more conservative calibration rule, it guarantees finite-sample SDR control under i.i.d. calibration and test observations, bounded losses, and almost surely distinct calibration scores.

Dear Reviewer,

We are looking forward to your feedback and would greatly appreciate any further questions or concerns you may have.

If there are additional issues that require clarification or further experiments, an earlier response would help us better address them within the remaining rebuttal period.

Thank you again for your time and consideration

Reviewer_z7c8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Uncertainty-Normalized Empirical Calibration (UNEC), a method for selecting promising candidates while controlling statistical risk. The main motivation comes from applications such as drug discovery, where machine learning models can screen many candidates, but experimentally validating every candidate is costly. UNEC is a p-value-free framework that directly calibrates a selection threshold using empirical losses on a held-out calibration set. This paper studies marginal discovery risk (MDR) and selective discovery risk (SDR), and establishes finite-sample MDR control for an adjusted rule and asymptotic SDR control with explicit limiting risk and power.

Critical strengths
  1. UNEC uses standardized predictive margins followed by empirical calibration and the selection procedure is relatively straightforward to implement, which is a simple and computationally efficient algorithm. Also, the paper provides meaningful finite-sample guarantee for MDR.
Critical weaknesses
  1. There is no finite-sample guarantee for SDR, which may limit the scope of the contribution.
Questions for authors

None.

3 comments
AuthorsOpenReview ↗

Response to Reviewer z7c8

Q: There is no finite-sample guarantee for SDR, which may limit the scope of the contribution.

A: We thank the reviewer for highlighting this limitation. We agree that finite-sample SDR control is desirable, particularly when the calibration sample is small. To address this concern, we have developed a new concentration-based variant of UNEC that achieves finite-sample SDR control for general bounded losses. This additional theoretical result is presented in Appendix B.4 of the revised manuscript.

This variant uses a more conservative calibration rule. Building on the population conditional risk Q(z)=E[L∣Z≥z]Q(z)=\mathbb{E}[L\mid Z\geq z] defined in Section 4.1, we construct simultaneous upper confidence bounds for Q(z)Q(z) across candidate cutoffs using an order-statistic argument, a KL-based concentration inequality, and a union bound.

Specifically, conditional on the training data and fitting randomness, we assume i.i.d. calibration and test observations, bounded losses L∈[0,1]L\in[0,1], and almost surely distinct calibration scores. We introduce an adjusted risk level qα=(q−α)/(1−α)q_\alpha=(q-\alpha)/(1-\alpha) for 0<α<q0<\alpha<q, together with a rank-dependent correction δk\delta_k to the empirical SDR calibration. The resulting selection rule satisfies SDR(S^SDR)≤q\mathrm{SDR}(\widehat S_{\mathrm{SDR}})\leq q in finite samples, without requiring conformal pp-values or ee-values.

We thank the reviewer again for this valuable observation, which motivated us to strengthen the theoretical analysis and clarify the available risk-control guarantees.

Dear Reviewer,

We are looking forward to your feedback and would greatly appreciate any further questions or concerns you may have.

If there are additional issues that require clarification or further experiments, an earlier response would help us better address them within the remaining rebuttal period.

Thank you again for your time and consideration

I appreciate the authors’ response. However, I believe my original assessment and score remain appropriate.

Reviewer_muctOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces UNEC (Uncertainty-Normalized Empirical Calibration), a statistical framework for selecting promising candidates from a large discrete pool, while controlling for the risk of selecting undesirable ones. Unlike previous similar works, the authors focus on directly using predictions and loss values of a held-out set without computing p-values. In this method, for each candidate, one predicts the outcome using a regression model μ^(x)\hat{\mu}(x) and normalizes the margin of the prediction from a specified threshold cc using the predictive uncertainty σ^\hat{\sigma}: μ^(x)−cσ^\frac{\hat{\mu}(x) - c}{\hat{\sigma}}. The candidates are ranked using this normalized score, and a held-out calibration dataset is used to determine how many of these candidates can be selected while controlling for a given empirical risk constraint. Empirical results on five simulated datasets and three molecular screening tasks show that the proposed method can achieve competitive or superior performance compared to evaluated baselines, at a much lower computational cost.

Critical strengths
  1. The proposed framework is simple, intuitive, and computationally efficient. Using only predictive modeling (for both mean and uncertainty estimation), the UNEC avoids the costly overhead of computing p-values (and e-values).
  2. The authors provide substantial theoretical support for the proposed method. Of note, the authors show finite-sample MDR control under conservative calibration, asymptotic SDR control, and asymptotic power optimality under a location-scale model for one-sided binary selection. They further show support for general bounded losses, observation-specific thresholds, and two-sided selection targets, demonstrating the flexibility of UNEC across different settings.
Critical weaknesses

Given the importance of uncertainty normalization to UNEC, as demonstrated by the UNEC-raw ablation performance compared to UNEC, one can argue that different uncertainty quantification methods can lead to different candidate rankings and, thus, affect selection power. Further experimentation would be beneficial in establishing the robustness of the framework to uncertainty quantification.

Questions for authors

Can the authors please demonstrate the robustness of their approach to different uncertainty quantification (UQ) methods by replicating some (if not all) of the results with alternative UQ methods? There are two questions here: (1) can UNEC continue controlling MDR or SDR when uncertainty is estimated differently, and (2) can UNEC retain its selection-power advantage when different uncertainty estimators are used?

3 comments
AuthorsOpenReview ↗

Response to Reviewer muct

We thank the reviewer for this suggestion. We have added scale-estimation sensitivity experiments in Appendix E (Figures 17 and 18) and address the two questions separately below, distinguishing risk-control validity from selection power.


Q1. Can UNEC continue controlling MDR or SDR when uncertainty is estimated differently?

Yes, provided the assumptions of the corresponding risk-control result hold. UNEC’s MDR and SDR guarantees do not require a particular uncertainty-estimation method or convergence of the estimated scale to the true conditional standard deviation.

Risk control is obtained through empirical calibration on held-out losses. As described in Section 3, the estimated scale enters the score Z(x;c)=(μ^(x)−c)/σ^(x)Z(x;c)=(\widehat{\mu}(x)-c)/\widehat{\sigma}(x), while the selection cutoff is calibrated separately. Conditional on the training data and fitting randomness, the fitted score is fixed, and the calibration and test observations remain i.i.d. Alternative scale estimators may change candidate rankings and therefore selection power, but the risk-control analysis applies to each fitted score that satisfies the corresponding theoretical conditions.

Specifically, Theorem 1 (Section 4.1) establishes asymptotic SDR control for a fitted score satisfying Assumption 1 (Appendix B.1). Theorem 2 (Section 4.1) guarantees finite-sample MDR control for the adjusted procedure with δ≥1−q\delta\geq 1-q under exchangeability and bounded losses, while Corollary 2 (Appendix B.3) provides an explicit finite-sample MDR bound for the unadjusted procedure.

The new sensitivity experiments in Appendix E (Figures 17 and 18) provide complementary empirical evidence. Across nine scale-estimation specifications in each setting, empirical SDR remains close to the nominal level in both simulated and real-data settings, with modest deviations at the smallest risk levels. These findings support the empirical stability of UNEC’s risk calibration across the evaluated scale estimators.


Q2. Can UNEC retain its selection-power advantage when different uncertainty estimators are used?

Yes, our additional experiments show that uncertainty normalization can retain substantial power gains over UNEC-Raw across the evaluated scale estimators, particularly at stringent risk levels, although the magnitude varies. The original ablation study in Section 5 (Figure 1) demonstrates the benefit of uncertainty normalization relative to UNEC-Raw, which uses the same mean predictor and empirical calibration but ranks candidates by the unnormalized margin.

To assess sensitivity to scale estimation, we evaluate three scale-learning models—random forests (RF), Gaussian processes (GP), and multilayer perceptrons (MLP)—and three residual-based scale targets under DGPs 3 and 4 (Figure 17, Appendix E). In DGP 3, a heteroscedastic location–scale model, the three residual targets yield similar power within each learning model, whereas differences across models are more apparent. A similar pattern appears in DGP 4, which departs from the location–scale structure. The following table presents representative results, using the conditional mean absolute residual as the common scale target. Each cell reports mean empirical SDR / mean power, both in percent, averaged over 500 repetitions.

SettingqqRFGPMLPUNEC-Raw
DGP30.1010.13 / 39.2910.08 / 28.5510.07 / 30.3710.42 / 26.20
DGP40.1010.19 / 38.4310.43 / 33.3410.38 / 24.9716.33 / 2.44

We extend this analysis to the AqSolDB molecular screening task (Figure 18, Appendix E) using LightGBM, XGBoost, and AttentiveFP for scale estimation. Part of the results, using the conditional mean absolute residual as the common scale target, are summarized as follow.

qqLightGBMXGBoostAttentiveFPUNEC-Raw
0.055.44 / 12.155.26 / 12.495.07 / 12.766.61 / 4.29
0.109.83 / 28.849.67 / 28.509.65 / 24.569.40 / 17.98

Selection power depends on the candidate ranking induced by the fitted scale. Theorem 3 (Section 4.2) establishes asymptotic oracle power optimality for a common one-sided binary target under the location–scale model and additional estimation-consistency conditions. This result does not guarantee oracle optimality for more general models or equal finite-sample power across uncertainty estimators.

Across the evaluated settings, power is relatively insensitive to the residual-based scale target within each learning model, whereas differences across scale-learning models are more noticeable, particularly at stringent risk levels. These findings identify the scale-learning model as an important practical choice for achieving favorable selection power, while the risk-control guarantees remain applicable under their respective theoretical assumptions. We have added this discussion to the revised manuscript.

Dear Reviewer,

We are looking forward to your feedback and would greatly appreciate any further questions or concerns you may have.

If there are additional issues that require clarification or further experiments, an earlier response would help us better address them within the remaining rebuttal period.

Thank you again for your time and consideration

I'd like to thank the author for providing additional results and clarifications. In particular, the results in Appendix E address my original concern regarding the robustness of UNEC. I also appreciate the distinction between risk-control validity and selection power. If I understood correctly, the former doesn't depend on the scale estimator, whereas the latter does. While the differences in power across the estimators are noticeable in some of the settings, I believe the results provide sufficient evidence that the benefits of uncertainty normalization are not specific to the original estimator. Thus, I consider my main concerns fully addressed and will maintain my positive score of 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.