acceptodds
Under review as a conference paper at ICLR 2027

FDR-Controlled Selection in Reinforcement Learning via Distributional Likelihood Ratios

Abstract

In high-stakes reinforcement learning, reliable deployment requires identifying instances where a target policy is safe or effective, rather than relying solely on average performance. We study the problem of selecting a subset of initial states whose returns exceed a given threshold, while controlling the false discovery rate (FDR) of erroneous selections. This problem is challenging due to unobservable long-horizon returns and distribution shift in off-policy settings. To address this, we propose a unified framework for FDR-controlled selection in RL based on distributional modeling and likelihood-ratio principles. Our approach estimates the return distribution under the target policy and constructs selection rules that directly quantify evidence for desirable outcomes, avoiding the need for explicit p-value construction. We further incorporate a temporal decomposition strategy and an offline calibration procedure to enable reliable FDR control under partial observability and policy mismatch. We provide theoretical guarantees and demonstrate empirically that our method achieves accurate FDR control with improved selection power across a range of environments.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, False Discovery Rate (FDR) Control, Selective Deployment, Distributional Reinforcement Learning, Likelihood Ratio, Off-Policy Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GMF1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies selective deployment of a fixed RL policy: selecting initial states whose future returns exceed a threshold while controlling the FDR of unsuccessful selections. It combines distributional return estimation with likelihood-ratio scoring and off-policy calibration. I find the problem interesting and the numerical results promising. My main reservation concerns the correctness and scope of the theoretical claims.

Critical strengths

The formulation connects FDR control to a meaningful RL deployment problem. Calibrating selection when target-policy long-horizon returns are unavailable is a worthwhile step beyond supervised selection. The framework accommodates different return estimators and is evaluated across three environments, including nonlinear control and simulated healthcare with 50 independent runs each. The experiments also show useful power gains.

Critical weaknesses

The main weakness is that the theoretical guarantees are not established as stated: the FDR calibration argument requires correction, the optimality claim is broader than the proof supports, and several necessary conditions are missing from the theorem statements. These issues affect the paper's central guarantees despite the promising empirical results.

Questions for authors
  1. The proof of Theorem 2 relies on an incorrect finite-batch FDR identity. Equation (14) on p. 25 equates FDR with the conditional failure probability among selected states. For a fixed statewise rule with selection probability b and conditional failure probability ρ, the correct identity is FDR = ρ{1−(1−b)^m}. Appendix C.7 states this identity correctly (p. 37), but the factor is absent from the optimality argument. The preceding calibration step on p. 25 likewise does not follow as written, since changing the nominal level generally changes b. Could the authors provide a corrected calibration argument based on the selection mass of the resulting rule, with appropriate weighting under off-policy sampling?

2.The claimed optimality class is broader than the proof establishes. Theorem 2 claims optimality among all FDR-controlling procedures (p. 8), but the proof compares limiting selection regions applied separately to each state (pp. 25–27). This does not establish optimality against procedures that jointly use the test batch or randomize their selections. Please specify the comparison class and revise the theorem accordingly; a correct optimality result over a clearly defined class would still be valuable.

3.The formal theorem statements omit necessary qualifications. The exact FDR equality in Theorem 1 requires an active calibration constraint, as acknowledged in Appendix C.7, but this condition is absent from the theorem (pp. 7–8). Correct specification and threshold convergence alone do not ensure equality: if all statewise failure probabilities lie between 0.01 and 0.02 and α=0.1, selecting every state remains strictly below the target. Theorem 2 also lists only Assumption 1, although its proof invokes convergence results that require weight stability. Please reconcile the statements, assumptions, and appendix discussion.

3 comments

Thank you for your highly detailed and constructive evaluation. Your precise identification of the omitted finite-batch factor (1−(1−b)m)(1-(1-b)^m) and the overly broad optimality claim was invaluable. We have thoroughly revised the methodology and theory to directly address all three of your concerns.

1. Correct finite-batch identity and calibration.

You were completely correct that the finite-batch identity FDR=ρ(1−(1−b)m)\text{FDR} = \rho(1-(1-b)^m) must explicitly account for the probability of making at least one selection, and our original proof failed to maintain this factor during optimization.

We have completely overhauled the calibration algorithm. For a measurable selection region AA, let νA=Pte(S0∈A)\nu_{A} = \mathbb{P}_{\text{te}}(S_{0} \in A) and aA=Pte(S0∈A,Gπ≤c)a_{A} = \mathbb{P}_{\text{te}}(S_{0} \in A, G^\pi \le c). The revised procedure directly estimates aAa_{A} and νA\nu_{A} across all candidate thresholds using self-normalized importance weights. It then maximizes the estimated success mass ν^n−a^n\hat{\nu}_{n} - \hat{a}_{n} subject to the exact finite-batch constraint:

a^nν^n(1−(1−ν^n)m)≤α\frac{\hat{a}_{n}}{\hat{\nu}_{n}} \left( 1 - (1 - \hat{\nu}_{n})^m \right) \le \alpha

By utilizing the identity aν(1−(1−ν)m)=a∑ℓ=0m−1(1−ν)ℓ\frac{a}{\nu}(1-(1-\nu)^m) = a \sum_{\ell=0}^{m-1}(1-\nu)^\ell, we mathematically ensure a continuous extension at zero mass, yielding uniform consistency of the finite-batch FDR criterion without dividing by zero. Our revised Theorem 2 rigorously establishes asymptotic FDR control and optimal power convergence for this corrected finite-batch framework.

2. Scope and proof of optimality.

We fully agree that the original optimality claim was broader than what the proof supported. In the revised Theorem 2, we have strictly narrowed the comparison class to rules of the form {j:Stest,j∈A}\lbrace j: S_{\text{test},j} \in A \rbrace, where AA is a fixed, deterministic, state-wise measurable region applied to every test instance independently. We no longer claim optimality over procedures that jointly adapt to the test batch or randomize selections.

Within this well-defined class, our proof now leverages a Neyman-Pearson logic: For any fixed target mass ν\nu, the region AνA_{\nu} (which selects the lowest ν\nu proportion of failure probabilities based on our likelihood-ratio score) strictly minimizes FDR and maximizes power. The continuous and monotonically increasing nature of the success mass ensures that our finite-batch calibrated threshold asymptotically identifies the global optimum within this class.

3. Equality conditions and assumptions.

Thank you for the excellent counter-example (failure probabilities between 0.010.01 and 0.020.02 with α=0.1\alpha=0.1). It perfectly illustrates why the exact equality in Theorem 1 requires an active calibration constraint. We have revised Theorem 1 to explicitly state that the limiting FDR is ρ(1−(1−ν)m)≤α(1−(1−ν)m)\rho(1 - (1 - \nu)^m) \le \alpha(1 - (1 - \nu)^m), and noted that exact equality with the nominal bound only occurs when the calibration constraint is strictly active (ρ=α\rho = \alpha).

Furthermore, we have consolidated all necessary qualifications. Revised Theorems 1 and 2 now explicitly invoke Assumptions 1-3 (Model Regularity, Weight Stability, and Calibration Sampling), explicitly distinguishing the active case (α<Ete[p(S0)]\alpha < \mathbb{E}_{\text{te}}[p(S_{0})]) from the trivial case where selecting the entire state space remains strictly feasible.

Just one small clarification: Algorithm 1 still shows the original calibration, while p. 15 explains how to switch to the finite-batch version. Please specify the calibration of the experiment in the final version. Point raised.

Thank you very much for following up and for reading the revision so carefully. Your earlier comments helped us correct the finite-batch argument and clarify the scope of Theorem 2. We also appreciate your pointing out that the experimental setup should make clear which calibration rule we used.

All reported results for our method use the original calibration in Algorithm 1 with the median threshold from Algorithm 2. On each repetition, we draw a subsample from the calibration replay buffer and compute a threshold η\eta using Algorithm 1. Repeating this step lets us use more of the available calibration data. We then apply the median of the resulting thresholds to the test batch.

Two considerations motivated this choice. First, our experiments use relatively large test batches. When the selection probability ν\nu is large enough that mνm\nu is appreciable, the factor 1−(1−ν)m1-(1-\nu)^m is close to one, and the two calibration criteria are similar. Second, the conditional failure probabilities used in calibration are estimated from offline data. We therefore preferred the more conservative criterion in Algorithm 1, which applies the nominal level α\alpha directly to the estimated conditional failure probability at each candidate threshold. The FDR bound in Theorem 1 retains the term Λ\Lambda to account for residual return-model approximation error.

The same likelihood-ratio ranking also underlies the finite-batch variant on p. 15. That variant incorporates the probability of making at least one selection when choosing the threshold. Theorem 2 shows that, under correct specification, this combination of ranking and calibration asymptotically attains the highest power among fixed statewise rules satisfying the finite-batch FDR constraint.

Following your suggestion, we will state in the Experimental Details section that Algorithm 1 determines the threshold for each replay subsample and Algorithm 2 combines these thresholds by taking their median. Thank you again for helping us make the experimental procedure clear.

Reviewer_CApXOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper considers the FDR-controlled selection problem in RL, aiming to select a subset of initial states whose returns exceed a given threshold while controlling the FDR of erroneous selections. The main farmework is to use data and subsampling to estiamte the conditional return cdf.

Critical strengths
  1. The formulation and setup are clearly presented
  2. The experiments are sufficient to illustrate the main framework
Critical weaknesses
  • I cannot clearly understand the proof of the critical theorem, Theorem 2. Especially, in line 1307, the authors used Theorem 1 to compute the asymptotic FDR as h(α)=α(1−(1−b)m)h(\alpha) = \alpha (1 - (1-b)^m). But with nominal level α\alpha, will the inner term bb depends on the level α\alpha and should be written as b(α)b(\alpha)? Please provide a clearer explanation regarding the proof.
  • The proof of Theorem 1 is also not well presented. I hope the authors can provide detailed roadmap regarding the proof. Especially, in page 37, the sentence "The theorem’s exact expression corresponds to an active calibration constraint". As far as I can understand, it requires additional assumption of Theorem 1.
Questions for authors
  1. the notation for b and πb\pi_b are misleading. please consider a better notation system
  2. I cannot find the definitionfor b appeared in Theorem 1. please provide more explainations
3 comments

We deeply appreciate your rigorous scrutiny of our theoretical results. You correctly identified a critical flaw in the original proof of Theorem 2 regarding how changing the nominal level alters the selection probability mass. We have completely restructured the calibration algorithm and rewritten Theorem 2 to resolve this issue rigorously, transforming it into a mathematically sound "Finite-Batch Calibration" framework.

1. Dependence of the selection probability on the nominal level.

You were absolutely right: the original two-step nominal level adjustment was flawed because modifying the threshold inherently changes the probability mass of the selected region. We have abandoned that approach.

In the revised manuscript, our finite-batch calibration directly estimates both the selection probability (ν^n\hat{\nu}_{n}) and the joint probability of selection and failure (a^n\hat{a}_{n}) at each candidate CDF threshold η\eta:

ν^n(η)=∑i,tw^itkI(F^(c∣Sit)≤η)∑i,tw^itk\hat{\nu}_{n}(\eta) = \frac{\sum_{i,t} \hat{w}_{itk} \mathbb{I}(\hat{F}(c \mid S_{it}) \le \eta)}{\sum_{i,t} \hat{w}_{itk}}

a^n(η)=∑i,tw^itkI(F^(c∣Sit)≤η)F^(G~it(c,k)∣Si,t+k)∑i,tw^itk\hat{a}_{n}(\eta) = \frac{\sum_{i,t} \hat{w}_{itk} \mathbb{I}(\hat{F}(c \mid S_{it}) \le \eta) \hat{F}(\tilde{G}_{it}(c,k) \mid S_{i,t+k})}{\sum_{i,t} \hat{w}_{itk}}

The algorithm now directly maximizes the estimated success mass ν^n(η)−a^n(η)\hat{\nu}_{n}(\eta) - \hat{a}_{n}(\eta) subject to the strict finite-batch FDR constraint:

a^n(η)ν^n(η)(1−(1−ν^n(η))m)≤α\frac{\hat{a}_{n}(\eta)}{\hat{\nu}_{n}(\eta)} \left( 1 - (1 - \hat{\nu}_{n}(\eta))^m \right) \le \alpha

Thus, the selection mass and failure probability are always perfectly coupled to the same candidate rule. The new Theorem 2 proves population optimality directly using a Neyman-Pearson framework and uniform convergence, entirely avoiding the flawed scaling trick.

2. The active-constraint condition in Theorem 1.

Thank you for pointing out the missing qualification. Correct specification and threshold convergence alone yield an inequality; exact equality requires the constraint to be active. In the revised Theorem 1, we explicitly define the limiting selection mass ν\nu and conditional failure rate ρ\rho, stating the limit as:

lim⁡n→∞FDR=ρ(1−(1−ν)m)≤α(1−(1−ν)m)\lim_{n \to \infty} \text{FDR} = \rho \left( 1 - (1 - \nu)^m \right) \le \alpha \left( 1 - (1 - \nu)^m \right)

We now clearly state that equality with the nominal bound only holds when the active calibration condition (ρ=α\rho = \alpha) is met.

3. Proof roadmap and notation.

Following your excellent suggestion, we have replaced the overloaded symbol bb with ν\nu to represent the selection probability throughout the text.

We have also completely rewritten the proof of Theorem 1 with a clear 5-step roadmap: (i) Derive the exact finite-batch FDR identity conditioned on the fitted rule; (ii) Establish uniform consistency of the calibration statistic; (iii) Bound the return-model error via Bellman contraction; (iv) Apply the empirical calibration constraint; and (v) Derive the asymptotic limit under correct specification. The supporting lemmas (Lemmas 4-6) now clearly trace the data flow from the full replay buffer to the weighted target averages under explicitly stated sampling assumptions (new Assumption 3).

Thank you for your time and efforts. I raise my score to 2

Thank you for reconsidering our submission and updating your assessment. We appreciate your careful review and constructive feedback, which helped us improve the theoretical statements and proofs.

Reviewer_DELgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a testing framework to risk-sentitive RL. Specifically, fix a policy pi, maximize wrt set of states expected proportion of states under which policy pi clears value threshold c that are selected, under an FDR constraint on discovered states.

Construct LR, statewise, of is s a state at which policy clears vs doesn't clear c. That produces a s-specific score. Apply a threshold t on scores so at to construct a selected set. Calibrated numerically t so as to achieve FDR constraint.

Use a replay based approach for calibration, using estimated pi-specific returns from via a subsampling device.

Critical strengths

Mostly clear exposition.

Theory not very wrong.

Numerics work.

Critical weaknesses

Why FDR? Risk sensitive RL is a well-established subfield with default baseline metrics such as cvar. Why is your criterion more desirable. You seem to pioneer the criterion in RL as much as the solution. It would make sense to justify this choice in much more depth.

You should credit past precedents of replay based calibration most prominently in the main text.

Questions for authors

Isn't the \hat F^\pi control very loose. Why not base it on MLE of dynamics at least in tabular case?

3 comments

We sincerely thank you for recognizing the clarity of our framework and the strength of our empirical results. Your insightful questions help us better articulate the motivation for FDR over traditional risk metrics, clarify our relationship to prior literature, and explain the role of return-distribution estimation.

1. Why FDR, and how does it relate to CVaR?

The central distinction lies in the object being evaluated. For a fixed policy, CVaR is a functional of the induced return distribution, typically used for policy optimization. In contrast, FDR evaluates a statistical selection procedure: it bounds the expected fraction of unsuccessful deployments among all instances selected by that procedure.

Let SD\mathcal{S}_{\mathcal{D}} be the selected deployment set based on offline data D\mathcal{D}. Our criterion is:

FDR=E[∑j∈SDI(Gjπ≤c)max⁡(1,∣SD∣)]\text{FDR} = \mathbb{E} \left[ \frac{\sum_{j \in \mathcal{S}_{\mathcal{D}}} \mathbb{I}(G_{j}^\pi \le c)}{\max(1, \lvert \mathcal{S}_{\mathcal{D}} \rvert)} \right]

Knowing a policy's population CVaR does not provide instance-level calibration for deployment. Even with a known MDP, deployment states and returns remain random. FDR perfectly complements risk-sensitive RL: a policy optimized via CVaR can subsequently undergo our FDR-controlled selection to guarantee that the proportion of failures in actual deployment remains below a strict safety budget. We have explicitly added this distinction to the Introduction.

2. Credit for replay-based calibration.

We fully agree that methodological antecedents should be prominently credited. In the revised manuscript, we have moved the discussion of Gan et al. (2025) directly into the main text (Preliminaries and Method sections), explicitly attributing the time-stratified replay sampling design to their work. Our specific novelty is built upon this sampling foundation: we utilize conditional return CDFs to construct likelihood-ratio conformity scores and calibrate them to achieve the FDR/Power objective under off-policy shift, addressing a distinct selective deployment problem.

3. Tightness of the return-model bound and tabular MLE alternative.

The return-model error term in our revised Theorem 1 is Λ=2LγkWˉ1(Fθ∗,Fπ)\Lambda = \sqrt{2L\gamma^k\bar{W}_{1}(F_{\theta^{\ast}}, F^\pi)}. This is a uniform error-propagation bound designed to rigorously quantify how residual distribution error affects FDR after a kk-step rollout. Under correct specification, this term naturally vanishes.

Regarding the tabular MLE: You are absolutely correct. In a tabular MDP, estimated transition probabilities and rewards can be directly propagated to obtain the return CDF, which fits perfectly into our framework. Our method is completely agnostic to the choice of the conditional return estimator. We utilized neural estimators (EMGN, IQN, Nadaraya-Watson) in the paper simply because our continuous-state experiments necessitated them. We have clarified in the revision that our likelihood-ratio selection and calibration framework readily accepts any valid return distribution estimator, including tabular MLE.

Dear authors, I have read your rebuttal but to be convinced that it makes sense to consider FDR I would have needed an application motivated justification. You simply contrasted properties of CVAR vs FDR but it still not clear why one set of properties is more desirbale than the other.

Dear Reviewer DELg,

Thank you for the clarification. We agree that the choice of criterion should follow from the application. Our claim is specific: when a deployment decision is governed by a tolerance threshold and a budget on the fraction of accepted jobs that may fail, FDR directly expresses the operational requirement. A general preference ordering between FDR and CVaR is unnecessary for this claim.

Consider a production line where, given a part’s initial pose, an operator decides whether to use a fixed robotic controller or send the part to manual handling. An automated placement outside the assembly tolerance requires one manual rework step. Suppose the operator’s requirement is to automate as many placements that would succeed as possible, while keeping the fraction requiring rework among automated placements below 5% on average across batches. The assembly tolerance determines what counts as a failure. The rework budget determines the denominator: only jobs assigned to the robot count. The resulting constraint is precisely FDR, and the objective is the power criterion studied in our paper.

The distinction has a practical consequence. Moving an already failed placement farther outside tolerance can worsen the CVaR of placement error, even though it still generates exactly one rework job. Conversely, consider a batch of 100 automated placements: moving five placements from just inside to just outside tolerance creates five additional rework jobs, however small the change in their placement errors. If large errors can damage equipment, their magnitude matters and CVaR is relevant. If the budget is for rework, the frequency of tolerance violations among automated jobs is the quantity the operator needs to control.

This is the application-based justification for our choice of FDR. We will include the example in the Introduction so that the deployment objective is explicit from the outset.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.