- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies off-policy prediction of a fixed policy’s finite-horizon return using independent trajectories collected by a behavior policy . The target policy samples i.i.d. actions from , selects a highest-scoring candidate, and optionally mixes this selection rule with . Given a fixed predictor , the objective is to construct a lower prediction bound with marginal coverage at least under the target policy.
The authors propose candidate completion (CC) method that adds fresh behavior actions around each logged action and uses candidate comparisons to accept or reject the logged trajectory, without collecting new environment outcomes or evaluating action densities. The retained trajectories have exactly the target-policy distribution, with expected sample count . Standard conformal calibration of their prediction residuals then provides the desired finite-sample marginal coverage guarantee.
The paper also develops weighted calibration and approximate-sampler extensions. Experiments on synthetic problems and MuJoCo tasks evaluate coverage and prediction loss against behavior-policy calibration, score-CDF weighting, and target-policy calibration.
Critical strengths
Strengths. The paper includes extensive empirical validation across controlled synthetic problems and multiple MuJoCo environments, with additional studies of tied scores, early termination, candidate budgets, and approximate behavior samplers. The breadth of these experiments reflects substantial implementation and evaluation effort. The tables and accompanying numerical results are generally well organized and visually clear, and the detailed supplementary results help readers inspect the method’s behavior across different settings.
Critical weaknesses
Beyond the strengths, I have concerns about the theoretical analysis, presentation, and practical value, organized into three main concerns below.
(a) Writing and problem formulation. The presentation does not adequately orient readers with a standard RL background to the problem being studied. The abstract introduces prediction errors, calibration, coverage, and best-of- policies before identifying the predicted outcome and the form of the returned bound, leaving readers to infer how these familiar terms fit together. In particular, the intended meaning of predictive coverage should be made explicit at its first appearance in the abstract, given the established use of “coverage” for behavior-target overlap in offline RL and OPE [1, 2]. The explanations subsequently provided in the introduction do not sufficiently clarify the relationship between this prediction task and familiar OPE objectives. The opening requires substantial revision to establish the research question and its scope before presenting the mechanism and experimental results.
Furthermore, Eq. 5 and 6 use the informal event labels “passes” and “retained” without displaying their mathematical definitions alongside the equations, requiring readers to assemble the event structure and the relevant randomness from the surrounding text and Algorithm 1. These examples illustrate a broader lack of precision and clarity in the presentation, and the paper therefore requires a substantially reorganized introduction and a more explicit problem formulation that establishes the objects, assumptions, and distinctions before presenting the method and its significance.
(b) Limited contribution and practical value. Sec. 3 requires to be “fixed before calibration” but imposes no accuracy or informativeness conditions. Thm. 2 therefore guarantees coverage even for an uninformative predictor, without ruling out practically vacuous bounds, see Eq. 7 that explicitly permits . The paper provides no theoretical characterization of how predictor error affects bound tightness or excess prediction loss, despite related conformal OPE work establishing asymptotic oracle approximation under known behavior policies and consistent conditional quantile estimation [3]. This gap between coverage validity and predictive usefulness is my central concern and the main reason for my low score. Unless the authors substantively address it (as well as other main concerns), I would maintain 1 (strong reject).
In addition, the paper provides no theoretical comparison of prediction-bound tightness between CC and other baselines. Its optimality result concerns acceptance rates within the restricted class of “exact rejection of unchanged whole trajectories” (App. C.5). Table 1 reports normalized pinball losses of for CC and for score-CDF weighting. The direct paired comparisons in App. B.3, Table 14, also report positive mean loss differences for CC relative to CDF at the minimum candidate budgets for , while Table 8 reports identical rounded losses of for weighted CC and score-CDF weighting at . Thus, the reported RL experiments do not demonstrate a consistent prediction-loss advantage over score-CDF weighting, and the corresponding theoretical tightness comparison remains absent.
(c) Missing discussion of modern OPE and distributional RL studies. Sec. 2 discusses conformal OPE but omits directly relevant work on distributional OPE and its connection to distributional RL [4]. In particular, Huang et al. [5] develop doubly robust return-CDF estimation with efficiency analysis, and Hong et al. [6] provide finite-sample error analysis for distributional OPE under function approximation.
The authors should explain how their residual-quantile calibration relates to these established objectives and compare the information requirements, approximation assumptions, and statistical guarantees. This omission leaves CC’s relationship to existing distributional OPE methods and its distinct contribution insufficiently articulated.
References
[1] Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., & Russell, S. (2023). Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism. arXiv [Cs.LG]. http://arxiv.org/abs/2103.12021
[2] Chen, J., & Jiang, N. (2019). Information-Theoretic Considerations in Batch Reinforcement Learning. arXiv [Cs.LG]. http://arxiv.org/abs/1905.00360
[3] Zhang, Y., Shi, C. & Luo, S.. (2023). Conformal Off-Policy Prediction. <i>Proceedings of The 26th International Conference on Artificial Intelligence and Statistics</i>, in <i>Proceedings of Machine Learning Research</i> 206:2751-2768 Available from https://proceedings.mlr.press/v206/zhang23c.html.
[4] Bellemare, M.G., Dabney, W. & Munos, R.. (2017). A Distributional Perspective on Reinforcement Learning. <i>Proceedings of the 34th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 70:449-458 Available from https://proceedings.mlr.press/v70/bellemare17a.html.
[5] Huang, A., Leqi, L., Lipton, Z. & Azizzadenesheli, K.. (2022). Off-Policy Risk Assessment for Markov Decision Processes . <i>Proceedings of The 25th International Conference on Artificial Intelligence and Statistics</i>, in <i>Proceedings of Machine Learning Research</i> 151:5022-5050 Available from https://proceedings.mlr.press/v151/huang22b.html.
[6] Hong, S., Qi, Z. & Wong, R.K.W.. (2025). Distributional Off-policy Evaluation with Bellman Residual Minimization. <i>Proceedings of The 28th International Conference on Artificial Intelligence and Statistics</i>, in <i>Proceedings of Machine Learning Research</i> 258:4006-4014 Available from https://proceedings.mlr.press/v258/hong25c.html.
Questions for authors
See weaknesses.
2 comments
I thank the authors for their response and the additional analysis. However, the revision does not give me sufficient grounds to raise my rating.
(a) The writing still needs substantial restructuring. I still find it difficult to follow a central argument through the rejection, clipping, stitching and pinball-risk results (Secs. 4–6, pp. 4–8), particularly as their assumptions and objectives change (Sec. 5.4, p. 7, lines 334–337, and Theorem 6, p. 7, lines 364–377). The revision often reads as a series of responses to individual objections, and this reinforces my impression of defensive writing from the authors without a sufficiently coherent account of their core insights. These points need to be worked into a coherent explanation of the problem, the key idea and why the subsequent results matter.
(b) I remain unsure what substantive new insight the combination provides. Still, several ingredients discussed in the paper have close precedents, including state-wise rejection and stitching in Mandel et al. [1] (Sec. 2, p. 3, lines 123–124), clipping corrections in Wang and Goel [2] (lines 117–120), and KL-dependent sampling costs in Chatterjee and Diaconis [3] (lines 124–127). Though the revision during rebuttal discusses these connections (Appendix F, p. 46, lines 2464–2479), I am still missing a convincing explanation of how the inherited ideas lead to something substantively new. A discussion organized around individual differences leaves the conceptual overlap and the significance of the combination unresolved.
(c) This brings me back to the central contribution. If the authors believe that their central contribution is correcting the distribution shift induced by best-of- selection from a fixed behavior policy through candidate completion and certified weighting (Sec. 1, p. 2, lines 72–92, and Lemma 1, p. 6, lines 270–281), the paper should focus on that contribution and present standard quantile calibration as an application (Theorem 2 and its rank argument, p. 5, lines 227–240). In that case, what makes the quantile-estimation component in this paper a substantive contribution beyond the distribution correction? Since the authors’ main contributions lie in candidate completion and its clipping certificate, the paper would tell a clearer story by centering this off-policy distribution correction, and perhaps the entire quantile-calibration component could even be removed to improve the flow of the paper.
If quantile estimation is instead central, the authors need to explain what specific structure of this RL problem enables an advance beyond this distribution correction. As currently presented, the pinball-risk decomposition appears to overlap substantially with those in existing work (Sec. 6, p. 8, lines 382–388, and Appendix F, p. 46, lines 2479–2481).
References
[1] Mandel, T., Liu, Y.-E., Brunskill, E., & Popović, Z. (2016). Offline Evaluation of Online Reinforcement Learning Algorithms. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). https://doi.org/10.1609/aaai.v30i1.10312
[2] Wang, J., & Goel, S. (2026). Weight Clipping for Robust Conformal Inference under Unbounded Covariate Shifts. arXiv [Cs.LG]. http://arxiv.org/abs/2605.02072
[3] Chatterjee, S., & Diaconis, P. (2017). The sample size required in importance sampling. arXiv [Math.PR]. http://arxiv.org/abs/1511.01437
Usefulness of the bounds: a new finite-sample risk bound, a predictor experiment, and a rewritten problem setup
Thank you for the detailed review. The revision resolves your central concern with a finite-sample risk bound (Theorem 6) and a predictor experiment (Table 2); we answer each point in order.
(a) Writing and problem formulation. The abstract now states the predicted quantity (the H-step return), the form of the output (the predictor minus a calibrated correction) and the guarantee before the method, and defines predictive coverage at first use. The introduction separates it from the overlap sense of coverage [1, 2]; Section 5.4 relates the two through a state-occupancy ratio. Section 3 defines the data, the target policy, the outcome and validity before the method. The events are written out: Equation 4 defines passes (U≤W/C), retained means Ut≤pt at every step, and Equations 6 and 7 display these events. The method names were reduced to three rules (rejection, weighted, certified).
(b) Validity versus usefulness. This was the main gap.
Theory (Theorem 6). Let εG be the excess pinball risk of the best constant shift of the predictor G over the ideal conditional quantile; it depends on G alone. With probability 1−δ:
Here λ is a lower bound on the error density over the band in which the estimated quantile can fall (stated in Theorem 6), and the error is assumed continuous with a finite mean. So calibration adds a term that vanishes with more data (up to the chosen β), and a predictor close to the conditional quantile yields a near-optimal bound. Part one is a finite-sample form of the asymptotic result of [3], without propensities.
Experiment (Table 2). We fit a quantile regression of the predictor's error on separate data and calibrate the adjusted predictor. On the 243 short-horizon evaluations this lowers the loss of the certified bound by 19% on average (in 98% of evaluations), and by 14–26% on the new long-horizon data, with coverage remaining at the nominal level; in the short-horizon experiment the calibrated bound stays within 0.6% of calibration on target data for either predictor.
Infinite corrections. Proposition 1 bounds the probability that the certified rule returns +∞ as a function of n, and Theorem 4 shows that with too few trajectories every valid procedure must be uninformative in the worst case. This also replaces the earlier optimality statement, which was restricted to rejection of whole trajectories: the lower bound now covers every valid procedure.
Tightness against score-distribution weighting (Proposition 5). Completion weights are the minimum-variance unbiased estimates of the ratio. The plug-in is biased for M≥3 at any fixed number R of competitors, so a weighted rule using it targets a different error distribution and its coverage error does not vanish with more logged data. The controlled sequential comparison shows this directly: with M=4 and R=3 the plug-in covers 81.4% over three steps and 76.5% over five, as the proposition predicts, while completion gives 90.0–91.4% with the same competitors. In the original MuJoCo comparison at the minimum budget (Table 24), completion is at or above the requested 90% in every setting (90.5%, 90.9%, 92.2% for M=4,8,16), where the plug-in gives 89.8%, 89.6% and 91.4%. At larger budgets the plug-in's bias is of order 1/R and the two losses are nearly identical, which explains the identical rounded losses you noted (Table 8 of the original submission, Table 22 in the revision). Completion therefore provides finite-sample validity at every competitor budget with comparable prediction loss.
(c) Distributional off-policy evaluation. Section 2 discusses [4], [5], [6]; Appendix F compares information requirements, assumptions and guarantees. In short: the estimators of [5] need behavior and target action probabilities (and a model for the doubly robust version) and give error bounds for the estimated return distribution; [6] fits the return distribution in a Markov model, with error bounds under realizability and a multi-step variant for the non-realizable case; completion needs only a sampler and gives finite-sample predictive coverage. Table 3 compares with the finite-sample importance-sampling constructions we could run (UnO band, OPRA IS-CDF band, PAC rejection sampling, weighted conformal rule): with exact ratios and 50,000 trajectories they return no bound at 15 and 20 steps, where the certified rule covers 92.2% and 96.6%.
We would welcome your assessment of whether these results address your concerns.