- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies selective deployment of a fixed RL policy: selecting initial states whose future returns exceed a threshold while controlling the FDR of unsuccessful selections. It combines distributional return estimation with likelihood-ratio scoring and off-policy calibration. I find the problem interesting and the numerical results promising. My main reservation concerns the correctness and scope of the theoretical claims.
Critical strengths
The formulation connects FDR control to a meaningful RL deployment problem. Calibrating selection when target-policy long-horizon returns are unavailable is a worthwhile step beyond supervised selection. The framework accommodates different return estimators and is evaluated across three environments, including nonlinear control and simulated healthcare with 50 independent runs each. The experiments also show useful power gains.
Critical weaknesses
The main weakness is that the theoretical guarantees are not established as stated: the FDR calibration argument requires correction, the optimality claim is broader than the proof supports, and several necessary conditions are missing from the theorem statements. These issues affect the paper's central guarantees despite the promising empirical results.
Questions for authors
- The proof of Theorem 2 relies on an incorrect finite-batch FDR identity. Equation (14) on p. 25 equates FDR with the conditional failure probability among selected states. For a fixed statewise rule with selection probability b and conditional failure probability ρ, the correct identity is FDR = ρ{1−(1−b)^m}. Appendix C.7 states this identity correctly (p. 37), but the factor is absent from the optimality argument. The preceding calibration step on p. 25 likewise does not follow as written, since changing the nominal level generally changes b. Could the authors provide a corrected calibration argument based on the selection mass of the resulting rule, with appropriate weighting under off-policy sampling?
2.The claimed optimality class is broader than the proof establishes. Theorem 2 claims optimality among all FDR-controlling procedures (p. 8), but the proof compares limiting selection regions applied separately to each state (pp. 25–27). This does not establish optimality against procedures that jointly use the test batch or randomize their selections. Please specify the comparison class and revise the theorem accordingly; a correct optimality result over a clearly defined class would still be valuable.
3.The formal theorem statements omit necessary qualifications. The exact FDR equality in Theorem 1 requires an active calibration constraint, as acknowledged in Appendix C.7, but this condition is absent from the theorem (pp. 7–8). Correct specification and threshold convergence alone do not ensure equality: if all statewise failure probabilities lie between 0.01 and 0.02 and α=0.1, selecting every state remains strictly below the target. Theorem 2 also lists only Assumption 1, although its proof invokes convergence results that require weight stability. Please reconcile the statements, assumptions, and appendix discussion.
3 comments
Just one small clarification: Algorithm 1 still shows the original calibration, while p. 15 explains how to switch to the finite-batch version. Please specify the calibration of the experiment in the final version. Point raised.
Thank you very much for following up and for reading the revision so carefully. Your earlier comments helped us correct the finite-batch argument and clarify the scope of Theorem 2. We also appreciate your pointing out that the experimental setup should make clear which calibration rule we used.
All reported results for our method use the original calibration in Algorithm 1 with the median threshold from Algorithm 2. On each repetition, we draw a subsample from the calibration replay buffer and compute a threshold using Algorithm 1. Repeating this step lets us use more of the available calibration data. We then apply the median of the resulting thresholds to the test batch.
Two considerations motivated this choice. First, our experiments use relatively large test batches. When the selection probability is large enough that is appreciable, the factor is close to one, and the two calibration criteria are similar. Second, the conditional failure probabilities used in calibration are estimated from offline data. We therefore preferred the more conservative criterion in Algorithm 1, which applies the nominal level directly to the estimated conditional failure probability at each candidate threshold. The FDR bound in Theorem 1 retains the term to account for residual return-model approximation error.
The same likelihood-ratio ranking also underlies the finite-batch variant on p. 15. That variant incorporates the probability of making at least one selection when choosing the threshold. Theorem 2 shows that, under correct specification, this combination of ranking and calibration asymptotically attains the highest power among fixed statewise rules satisfying the finite-batch FDR constraint.
Following your suggestion, we will state in the Experimental Details section that Algorithm 1 determines the threshold for each replay subsample and Algorithm 2 combines these thresholds by taking their median. Thank you again for helping us make the experimental procedure clear.
Thank you for your highly detailed and constructive evaluation. Your precise identification of the omitted finite-batch factor (1−(1−b)m) and the overly broad optimality claim was invaluable. We have thoroughly revised the methodology and theory to directly address all three of your concerns.
1. Correct finite-batch identity and calibration.
You were completely correct that the finite-batch identity FDR=ρ(1−(1−b)m) must explicitly account for the probability of making at least one selection, and our original proof failed to maintain this factor during optimization.
We have completely overhauled the calibration algorithm. For a measurable selection region A, let νA=Pte(S0∈A) and aA=Pte(S0∈A,Gπ≤c). The revised procedure directly estimates aA and νA across all candidate thresholds using self-normalized importance weights. It then maximizes the estimated success mass ν^n−a^n subject to the exact finite-batch constraint:
ν^na^n(1−(1−ν^n)m)≤α
By utilizing the identity νa(1−(1−ν)m)=a∑ℓ=0m−1(1−ν)ℓ, we mathematically ensure a continuous extension at zero mass, yielding uniform consistency of the finite-batch FDR criterion without dividing by zero. Our revised Theorem 2 rigorously establishes asymptotic FDR control and optimal power convergence for this corrected finite-batch framework.
2. Scope and proof of optimality.
We fully agree that the original optimality claim was broader than what the proof supported. In the revised Theorem 2, we have strictly narrowed the comparison class to rules of the form {j:Stest,j∈A}, where A is a fixed, deterministic, state-wise measurable region applied to every test instance independently. We no longer claim optimality over procedures that jointly adapt to the test batch or randomize selections.
Within this well-defined class, our proof now leverages a Neyman-Pearson logic: For any fixed target mass ν, the region Aν (which selects the lowest ν proportion of failure probabilities based on our likelihood-ratio score) strictly minimizes FDR and maximizes power. The continuous and monotonically increasing nature of the success mass ensures that our finite-batch calibrated threshold asymptotically identifies the global optimum within this class.
3. Equality conditions and assumptions.
Thank you for the excellent counter-example (failure probabilities between 0.01 and 0.02 with α=0.1). It perfectly illustrates why the exact equality in Theorem 1 requires an active calibration constraint. We have revised Theorem 1 to explicitly state that the limiting FDR is ρ(1−(1−ν)m)≤α(1−(1−ν)m), and noted that exact equality with the nominal bound only occurs when the calibration constraint is strictly active (ρ=α).
Furthermore, we have consolidated all necessary qualifications. Revised Theorems 1 and 2 now explicitly invoke Assumptions 1-3 (Model Regularity, Weight Stability, and Calibration Sampling), explicitly distinguishing the active case (α<Ete[p(S0)]) from the trivial case where selecting the entire state space remains strictly feasible.