- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the problem of candidate screening by proposing a p-value-free approach based on a score that combines the predicted value and its estimated variance. The score measures the normalized distance of each candidate from a decision boundary. A calibration procedure is then used to control selective discovery risk (SDR) and marginal discovery risk (MDR).
The theoretical contributions include finite-sample MDR control, asymptotic SDR control, and an analysis of statistical power. The authors further establish power optimality under a location-scale model. The proposed method is evaluated on diverse synthetic and real-world datasets.
Critical strengths
-
Clear presentation and practical efficiency. The paper is well written, and the proposed method is simple to implement and computationally efficient. Its p-value-free formulation makes it an appealing alternative to conventional screening procedures.
-
Theoretical analysis and empirical validation. The paper provides theoretical guarantees for MDR and SDR control, together with a statistical power analysis. The theoretical results are complemented by experiments on diverse synthetic and real-world datasets, supporting the practical relevance of the approach.
Critical weaknesses
- Limited novelty and generality of the theoretical results. The proposed calibration procedure appears closely related to existing conformal risk control methods, and the paper does not sufficiently clarify its methodological novelty relative to these approaches beyond the setting difference. Moreover, theory only establishes asymptotic SDR control. Furthermore, the power optimality result is established under a restrictive location-scale model that appears particularly suited to the proposed score. It is unclear how realistic this assumption is or whether the optimality result extends to more general settings.
- Limited empirical advantage in statistical power. In many experimental settings, the proposed UNEC method achieves lower power than the SCoRE baseline. This raises concerns about its practical benefits and whether the proposed approach offers a meaningful improvement over existing methods.
Questions for authors
-
The calibration procedure appears similar to existing conformal risk control approaches. Could the authors clarify the key methodological differences and identify which aspects of the proposed framework are fundamentally new beyond the difference in the setting?
-
The power optimality result relies on a specific location-scale model that appears tailored to the proposed score. How realistic is this model for the applications considered? To what extent does the optimality result remain valid under deviations from these assumptions?
-
The paper considers three main baselines: CS, NP, and SCoRE. However, some experiments compare UNEC against CS and NP, while others compare it against SCoRE. Could the authors provide consistent comparisons against all three baselines across the experiments?
-
Why is the benefit of UNEC more pronounced for small q values?
5 comments
Response to Reviewer A7jR (Part II)
Q3. The paper considers three main baselines: CS, NP, and SCoRE... Could the authors provide consistent comparisons against all three baselines across the experiments?
CS and NP are designed for FDR control under binary false-selection loss and do not directly provide SDR control for general bounded losses. This is why we compare UNEC with CS and NP in Figure 1, where SDR reduces to FDR, and with SCoRE in Figure 2, where the experiments involve general bounded losses and require SDR control.
These comparisons are also connected. Corollary 5.4 of Bai and Jin (2026) establishes that a binary-loss specialization of SCoRE is equivalent to clipped CS under the same underlying score and appropriate parameter settings. Thus, our CS comparison also covers this particular specialization of SCoRE.
Overall, the choice of baselines reflects their respective risk-control capabilities rather than an omission of relevant comparisons.
Q4. Why is the benefit of UNEC more pronounced for small q values?
We attribute UNEC's observed power advantage at small to the combined effects of uncertainty-aware ranking and direct empirical risk calibration, rather than to a universal advantage over conformal methods.
First, UNEC accounts for local predictive uncertainty when ranking candidates, distinguishing candidates with similar predicted means. Theorem 3 (Section 4.2) justifies this ranking under the location-scale model. Empirically, Appendix C.3 (Figure 6) demonstrates favorable power gains at comparable empirical FDR levels, while the additional experiments in Appendix E (Figure 18) show that the improvement over UNEC-Raw persists across different uncertainty estimators, particularly at small .
Second, UNEC selects the score cutoff directly from the empirical loss curve (Algorithm 1), whereas CS and SCoRE rely on conformal -values or -values followed by BH or e-BH, whose rejection requirements may limit discoveries under stringent risk constraints. This direct calibration can permit less conservative selections, particularly at small , when the candidate ranking and empirical risk estimates are sufficiently informative.
Together, uncertainty-normalized ranking and direct empirical calibration help explain UNEC's observed power advantage at small . This advantage is empirically supported and task-dependent, as illustrated by the additional Merck experiments in Appendix D.3, rather than a universal guarantee.
W1. Limited empirical advantage in statistical power.
Neither UNEC nor SCoRE consistently outperforms the other in statistical power across tasks. However, UNEC achieves competitive power with substantially lower computational cost, making it a practical alternative for large-scale risk-controlled selection.
We added two molecular screening benchmarks, 3A4 and LOGD, from the Merck Molecular Activity Challenge (Appendix D.3, Figures 15–16). The results show task-dependent power differences: SCoRE-homo achieves higher power on 3A4 at comparable empirical SDR, whereas UNEC achieves higher power than all three SCoRE variants on LOGD while maintaining empirical SDR close to the nominal level. Notably, UNEC also requires substantially less computation on both benchmarks, as shown below.
3A4
| UNEC | SCoRE-dtm | SCoRE-homo | SCoRE-hete | |
|---|---|---|---|---|
| 0.10 | 9.82 / 27.28 | 0.78 / 2.90 | 9.52 / 37.20 | 10.73 / 25.70 |
| 0.20 | 20.06 / 70.66 | 4.02 / 15.86 | 20.06 / 79.07 | 19.89 / 72.23 |
| Runtime (s) | 15.11 | 307.32 | 307.32 | 307.32 |
LOGD
| UNEC | SCoRE-dtm | SCoRE-homo | SCoRE-hete | |
|---|---|---|---|---|
| 0.10 | 8.37 / 8.77 | 1.10 / 0.94 | 6.51 / 5.93 | 6.35 / 3.69 |
| 0.20 | 19.53 / 30.28 | 4.22 / 5.70 | 18.05 / 24.70 | 17.76 / 17.33 |
| Runtime (s) | 18.40 | 266.38 | 266.38 | 266.37 |
Each risk-level cell reports mean empirical SDR / mean power, both in percent. Runtime is averaged per repetition over the full grid of , including model fitting, prediction, and selection.
W2. Lack of finite-sample SDR control.
We have addressed this limitation by developing a conservative, concentration-based variant of UNEC that achieves finite-sample SDR control for general bounded losses (Appendix B.4). This extends the theoretical guarantees of UNEC beyond the asymptotic SDR control established for the standard procedure (Theorem 1).
Specifically, the new variant constructs simultaneous upper confidence bounds for the conditional risk using an order-statistic argument, a KL-based concentration inequality, and a union bound. With an adjusted risk level for and a more conservative calibration rule, it guarantees finite-sample SDR control under i.i.d. calibration and test observations, bounded losses, and almost surely distinct calibration scores.
Dear Reviewer,
We are looking forward to your feedback and would greatly appreciate any further questions or concerns you may have.
If there are additional issues that require clarification or further experiments, an earlier response would help us better address them within the remaining rebuttal period.
Thank you again for your time and consideration
Response to Reviewer A7jR (Part I)
Thank you very much for your feedback:
The main contribution of
UNECis the combination of uncertainty-normalized ranking and direct empirical risk calibration, rather than the introduction of a different risk-control objective. As described in Section 3 (Algorithm 1),UNECranks candidates using the standardized predictive margin Z(x;c)=(μ(x)−c)/σ(x) and determines the selection cutoff directly from held-out losses, without constructing conformal p- or e-values.Computationally,
UNECcalibrates selection thresholds using sorted scores and cumulative losses, whereas the evaluatedSCoREimplementations require repeated conformal e-value calculations across candidate thresholds and test observations. As discussed in Section 5, these calculations dominateSCoRE’s selection cost, with computational complexity scaling approximately as m(n+m) in our implementation. This difference is reflected in Figure 2, whereUNECrequires approximately 1.3 seconds per repetition, compared with 35–37 seconds forSCoREover the full grid of q.Also, existing approaches differ in their constructions and guarantees. Conformal risk control (
CRC; Angelopoulos et al., 2024) calibrates predictive procedures to control expected loss under a monotonicity condition, rather than directly controlling the average loss among selected observations.CS(Jin and Candès, 2023) combines conformal p-values with multiple testing to control FDR, whileSCoRE(Bai and Jin, 2026) uses risk-adjusted conformal e-values for general selective risk control. TheNPmethod (Qin et al., 2025) uses likelihood-ratio-based ranking with threshold calibration. In contrast,UNECconstructs its ranking from estimated conditional location and scale and directly calibrates empirical MDR or SDR.A key strength of
UNECis its broad scope and flexibility. The framework supports general bounded losses beyond binary false-selection errors, allowing selection risk to reflect graded errors or application-specific costs. By adapting the standardized margin and corresponding loss, the same calibration procedure readily accommodates common one-sided targets, observation-specific thresholds, and two-sided interval targets, without requiring a separate testing procedure for each setting (Section 3andAppendix A).The location–scale assumption provides a sufficient condition for the power optimality of UNEC, rather than a restriction on its applicability or risk-control guarantees.
The location–scale family is substantially broader than the Gaussian model. It accommodates nonlinear conditional means, heteroscedasticity, and general continuous error distributions, provided the standardized error distribution is common across covariate values. Under this model, Section 4.2 establishes that
=1-F_\epsilon\left(-\frac{\mu(x)-c}{\sigma(x)}\right).$$ Hence, the oracle standardized margin preserves the ordering of the posterior success probability without requiring Gaussian errors or explicit estimation of the residual distribution. We acknowledge that requiring a common standardized error distribution is a restriction. Outside this family, conditional distributions may change shape across covariates, and their first two moments do not generally determine the posterior success probability. The standardized margin may then fail to produce the optimal ranking. Nevertheless, `UNEC` remains applicable because its risk calibration does not depend on this assumption. In particular, `Theorems 1 and 2` establish risk-control guarantees under the corresponding calibration, exchangeability, and regularity conditions, without requiring a location–scale model. Our empirical results further examine deviations from this structure. `DGPs 4–5` (Appendix C.2) are covariate-dependent Gaussian mixtures that fall outside the location–scale family, **yet `UNEC` retains competitive risk–power performance (`Figures 1–2`)**. Moreover, our new sensitivity analysis in `Appendix E (Figure 17)` evaluates nine scale-estimation specifications. In DGP 4, at $q=0.10$, uncertainty-normalized scores achieve `24.97%–38.84% power`, compared with `2.44%` for `UNEC-Raw`, while attaining empirical SDR closer to the nominal level. These findings support UNEC’s empirical usefulness beyond the location–scale family. Although UNEC delivers strong power in practice, these results do not establish universal power optimality under arbitrary conditional distributions. Extending the optimality result to such distributions remains an open theoretical question.