acceptodds
Under review as a conference paper at ICLR 2027

crBH: Conformal Top- Selection with False Discovery Rate Control

Abstract

Selecting candidates that truly belong to the top-k of a large pool is important in scientific screening and information retrieval, but their outcomes are unavailable at selection time. Existing conformal selection rules address fixed outcome thresholds; top-k membership instead depends on the other items in the test pool. Calibration and test ranks also live on different scales when their pool sizes differ. We formulate inductive top-k selection with false discovery rate (FDR) as its error criterion and propose crBH to address this rank-space mismatch. An exact rank projection supplies calibration-item probabilities of falling outside the test topk; projected conformal statistics combine these probabilities with trained ranking scores. A null-proportion-adjusted BH step-up rule then selects a score-ordered subset. We characterize the asymptotic FDR and power of crBH and connect its power to the quality and ordering induced by the ranking model. Although the guarantee is asymptotic, controlled and application benchmarks show empirical FDR close to the nominal level at the studied finite sample sizes when the method makes substantial discoveries, while retaining substantial power. For applications requiring an exact finite-sample guarantee, we additionally give a conservative certified variant under joint calibration/test exchangeability.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, rank selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fM5COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper develops a conformal framework for top-kk selection with false discovery rate (FDR) control. It constructs exact rank-projection weights using auxiliary rank constructions and incorporates them into a BH-type selection procedure, avoiding the need to specify an additional outcome threshold. The paper theoretically characterizes the asymptotic FDR, power, and oracle ranking properties of crBH, while Certified-crBH provides a finite-sample FDR guarantee under joint exchangeability and prespecified parameters.

Critical strengths
  • The paper uses exact rank-projection probabilities as calibration weights to construct a conformal procedure for top-kk selection with FDR control. Unlike fixed-threshold conformal selection, this addresses an unknown cutoff determined jointly by the test outcomes.

  • The analysis establishes asymptotic FDR and power results, including conditions for attaining oracle power within the specified rule class. Certified-crBH additionally provides a finite-sample FDR guarantee under joint exchangeability.

Critical weaknesses

The main asymptotic analysis in Theorem 5 assumes k/m→τ∈(0,1)k/m\to\tau\in(0,1), and therefore does not cover the sparse-selection regime k/m→0k/m\to0, including fixed kk as m→∞m\to\infty. Although Certified-crBH provides finite-sample FDR control, its minimum nonempty selection size can severely restrict power when kk is small and calibration and test sizes are comparable. The paper does not provide a corresponding power characterization in this important regime.

Questions for authors
  • How does crBH differ from conformal FDR control after data-dependent selection [1] and FDR-controlled ranking inference under the BTL model [2]? Which differences in the hypotheses, observed data, and assumptions prevent their direct application here?

  • Appendix C.2 compares the exact weights with a mass-matched scaled-rank step approximation using the same information and selection procedure. Could the asymptotic FDR/power results and the oracle-attainment claim also hold with these simpler weights under the same assumptions? If not, which step of the proof specifically requires the exact projection?

  • Corollary 6 allows randomization at the oracle threshold, and Section 3 uses randomized score tie-breaking. However, the power bound in Appendix B.1 and Appendix K uses the strict sets S>s{S>s} and ητ>c{\eta_\tau>c}, which exclude boundary points. How does the bound account for candidates selected through randomization at these boundaries?

  • Lines 428 and 433 refer to "Appendix 4". Are these references intended to point to Figure 4 in Appendix C.1?

[1] Wang, X., Huo, Y., Peng, L., and Zou, C. Conformalized Multiple Testing after Data-dependent Selection. NeurIPS, 2024.

[2] Liu, Y., Fang, E. X., and Lu, J. Lagrangian Inference for Ranking Problems. Operations Research, 2022.

Reviewer_tPRgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies the Top-K selection problem using conformal inference, focusing on selecting Top-K candidates in unlabeled test data. To address this problem, the authors proposed an exact rank projection to construct weighted conformal statistics, and then used the corrected BH procedure to perform the selection. Asymptotic bounds on FDR and power are established.

Critical strengths
  1. The Top-K selection problem studied in this paper is different from existing conformal selection work, which is interesting.

  2. The exact rank projection method is novel in conformal inference.

Critical weaknesses
  1. This paper is not well written, making it difficult to follow and understand the construction. E.g., Section 3 should focus on the problem setup; however, the authors suddenly begin to discuss the pretrained score, which is part of the method, not the problem setup. Also, Remark 1 is confusing and refers to crBH, which this section does not introduce. For the central part of this paper, Section 4.1 is extremely difficult to follow.

  2. There are many notations that are not clearly defined, and I have to guess their definitions. For example, what are YicY_i^c and YitY_i^t? I guess they stand for the calibration label and test label. But they conflict with the notations {(Xi,Yi)}i=1n+m\{(X_i,Y_i)\}_{i=1}^{n+m}. Also, RjtR_j^t is not defined. Similar issues for R~i\widetilde R_i, "competitor labels".

  3. The conditions to achieve asymptotic FDR control in Theorem 5 are not deeply discussed. When will there be a unique positive sign-changing crossing?

  4. The conclusion in Proposition 7 is weak, since it restricts the sample size.

Questions for authors

Is rank-projection necessary? What if we set qi∗=1q_i^* = 1 in Eq. (7)?

Reviewer_fx6WOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies selecting candidates that belong to the true top-(k) of a test pool while controlling the false discovery rate. The proposed method, crBH, uses a combinatorial rank projection to convert calibration ranks into fractional null-reference weights, constructs weighted score-comparison statistics, and applies a null-proportion-adjusted BH procedure. The authors establish asymptotic FDR and power results and provide a conservative certified variant with finite-sample FDR control. Experiments evaluate the methods across synthetic data and three application datasets.

Critical strengths
  1. The paper addresses a meaningful selection problem where success depends on a candidate’s rank relative to the other test candidates.
  2. The rank-projection construction is reasonably simple and provides a principled way to account for different calibration and test pool sizes.
  3. The theoretical analysis distinguishes the asymptotic validity of raw crBH from the finite-sample validity of the certified procedure. The explicit counterexample to universal finite-sample control and the experiments quantifying the cost of certification help clarify the methods’ limitations.
Critical weaknesses
  1. Limited comparison with existing calibrated selection methods. The main comparisons are against uncalibrated model top-(k) selection. In particular, Recommendation Systems with Distribution-Free Reliability Guarantees (Angelopoulos et al., 2023) already uses Learn-then-Test to control false discoveries defined by exclusion from a true top-ranked subset. Its calibration unit is a fully ranked candidate list, which differs from this paper’s individual-item calibration. However, this distinction should be discussed explicitly, and a comparison where sufficient calibration pools are available would help establish crBH’s advantages in power and calibration efficiency.
  2. Weak evidence of practical screening utility. Virtual screening is a natural application, but there is no activity-based screening benchmark. The ESOL experiment evaluates solubility ranking and shows limited recovery: at (\alpha=0.1), the best certified model recovers only 3.53% of the true top-(k), selecting approximately 4.1 molecules on average. These results leave the practical usefulness of the method unclear.
  3. The MIMIC-IV use case is insufficiently motivated. It is unclear what actionable clinical decision is supported by selecting patients in the longer half of remaining ICU stays. The paper should explain why relative top-(k) membership is the appropriate target and how controlling false discoveries for this target would improve resource allocation.
  4. The main procedure has a qualified asymptotic guarantee. The result requires a stable positive selection boundary and (k/m) converging to a nonzero fraction, excluding the fixed-small-(k) regime relevant to some screening budgets. The certified variant addresses finite-sample validity but can substantially reduce power. This tradeoff deserves greater emphasis in the main presentation.
  5. There are severe reproducibility concerns with the paper:
    • Missing experimental settings: The main ESOL and Yummly experiments do not specify kk; Yummly and MIMIC also lack exact split sizes. ESOL fingerprint parameters and complete architectures, hyperparameters, and training protocols for the ranking models are not specified. The manuscript does not specify split construction (patient-level versus stay-level splitting), resampling procedures for MIMIC.
    • Code availability: The linked code is expired and does not point to a valid code link.
Questions for authors
  1. How does crBH compare with the Learn-then-Test approach of Angelopoulos et al. (2023), which also defines false discoveries through true top-ranked membership? Could you compare power and calibration requirements when enough labeled data are available to construct calibration pools?
  2. How are the calibration and test splits constructed in each experiment? For ESOL specifically, are these random splits or predefined benchmark splits, and how does the construction support joint exchangeability?
  3. Could you evaluate the method on a virtual screening task with measured biological activity or binding affinity? Such an experiment would help establish whether the method provides useful recovery under realistic screening budgets.
  4. Could you address all the reproducibility concerns ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.