- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
-
Proposes InCOPE, a prior-data fitted network for off-policy evaluation (OPE), pretrained on synthetic MDPs with exact target-policy values obtained via dynamic programming.
-
Uses behavior-policy trajectories, target-policy information, and estimated transition dynamics to predict policy values without task-specific parameter updates.
-
Evaluates the method across four tabular domains, reporting improvements over conventional OPE estimators and conducting ablations on pretraining priors and current-task information.
Critical strengths
-
Meaningful problem: Leveraging cross-task knowledge to improve OPE under limited data and substantial policy mismatch is a valuable research direction.
-
Promising empirical results: InCOPE consistently achieves lower MAE than the evaluated baselines across four domains. The paper is clearly presented, with detailed methodology and useful ablations.
Critical weaknesses
W1. The experiments do not sufficiently establish the benefit of the proposed estimator beyond the use of additional cross-task prior information. InCOPE is pretrained on synthetic MDPs with exact policy-value labels, whereas the six main OPE baselines only use current-task trajectories. This comparison is reasonable for demonstrating the benefits of pretraining, but cannot isolate whether the improvements arise from the proposed direct policy-value prediction mechanism or simply from access to additional task information. Since the source tasks are generated from known MDPs, the same source information could be used to learn transferable reward/transition models or Bayesian priors, followed by model-based evaluation using current-task observations. Approaches with clearer and more careful ablations would provide a more direct comparison of how prior information should be incorporated into OPE.
W2. Existing controls are insufficient: Section 5.5 compares InCOPE against Ridge, kNN, RBF-SVR, and ExtraTrees using complete task inputs. However, these regressors operate on a 98,228-dimensional flattened representation with only 3,000 source-training queries per domain (Appendix C.6). They do not provide a strong alternative that explicitly exploits the MDP structure or Bellman equations. Consequently, the results establish the usefulness of the overall pretrained framework more convincingly than the distinctive advantage of InCOPE's estimator design.
W3. The robustness and transferability of the synthetic prior are insufficiently characterized, limiting the practical significance of the approach.
-
W3.1. Practical implications of theoretical assumptions: Proposition 1 characterizes the Bayes-optimal predictor under the pretraining prior, while Corollary 2 establishes optimality under conditional-mean compatibility between training and test distributions. Since exact compatibility is unlikely in practice, it would be more valuable to understand how violations of this assumption affect estimation accuracy and real-world applicability.
-
W3.2. Insufficient evaluation of prior mismatch: Section 5.4 modifies source reward distributions and policy ranges, demonstrating that the pretraining prior matters. However, it does not systematically examine performance degradation as test-time reward mechanisms, transition dynamics, or policy-generation mechanisms deviate from the source prior.
-
W3.3. Unclear conditions for successful transfer: The paper does not establish which aspects of task similarity matter most, how prior mismatch interacts with data availability or coverage, or how much mismatch InCOPE can tolerate before its advantage disappears or negative transfer occurs. These findings would provide important practical guidance for deploying pretrained OPE estimators.
Questions for authors
Q1. Alternative use of source-task information.
Could the authors compare InCOPE against a prior-aware model-based OPE approach trained using the same source MDPs and supervision? For example, learning transferable reward/transition priors followed by current-task model estimation and dynamic programming would help distinguish the benefit of cross-task knowledge from that of direct policy-value prediction.
Q2. Sensitivity to prior mismatch.
Could the authors provide controlled experiments that progressively vary the similarity between source and test MDPs, separately considering reward functions, transition dynamics, and policy-generation mechanisms?
In particular, at what degree of mismatch does InCOPE lose its advantage over current-task-only OPE estimators, and how does this threshold depend on the number of available trajectories? Such results would make the method's applicability much clearer.
Q3. Oracle-derived target policies.
In Appendix C.1, target policies in several domains are constructed from true-MDP action values, such as (Q_0^u). Although these values are not directly provided to InCOPE, the resulting policy probabilities can encode information about the underlying environment.
Would the reported improvements persist when evaluating target policies generated independently of the true MDP parameters, such as randomly generated policies or policies learned exclusively from offline data? This would help determine the extent to which performance depends on the benchmark's policy-construction procedure.
Q4. Long-horizon value modeling.
Section 4.2 uses only two learned propagation blocks, while Appendix B.1 already computes empirical target-policy occupancy statistics through multi-step transition propagation.
Could the authors disentangle the contribution of these precomputed occupancy features from learned propagation, and evaluate how performance changes with horizon and propagation depth? This would clarify whether the model learns transferable long-horizon value relationships or primarily exploits information already available in the engineered inputs.
5 comments
Q3. Target policies constructed without oracle information.
To address this concern, we evaluated the frozen Clinical specialist on 29 target policies constructed without using the true transition, reward or Q functions: one uniform policy, 25 fixed action-biased policies (90% on a designated action plus 10% uniform exploration), and three state-dependent random policies. The Clinical MDPs were unchanged, and InCOPE was not retrained.
Averaging over these policies, H=5/10/20, n=72/96/120 and five training seeds, InCOPE achieves MAE 0.0576, compared with 0.0654 for the best of the four source-validation-tuned compact regressors. Thus, the advantage in these Clinical settings is also observed when target policies are generated independently of the true MDP parameters. The detailed setup and comparison are in our response to Reviewer C2Vv, Q4 (Table C4). If accepted, we will include this evaluation in the paper.
Q4 / W2. Occupancy features, learned propagation and compact controls.
Removing learned propagation increases mean MAE in all four original test domains, even with occupancy inputs retained; the best depth among L=1,2,4 varies by domain. Our occupancy summaries are auxiliary inputs motivated by established OPE approaches that estimate target-policy visitation from offline data [1,2]. Their removal improves Dispatch but increases mean error in the other three domains. These findings support treating them as optional inputs, with the no-occupancy model as an alternative configuration. The detailed ablation results are provided in our response to Reviewer C2Vv, Q4 (Table C3).
We further examined horizon dependence using 50 paired evaluation queries per domain, fixing the MDP, policies and trajectory count and using nested trajectory prefixes. Below are selected horizon settings, including extensions beyond the source-training range. Values are specialist MAEs averaged over five training seeds, in the listed H order.
Table U4. Horizon and propagation ablations (MAE).
| Domain | H | Full (L=2) | No propagation (L=0) |
|---|---|---|---|
| GridWorld | 12 / 24 / 48 | 0.300 / 0.897 / 1.416 | 0.348 / 1.100 / 1.715 |
| Tumor | 25 / 50 / 100 | 0.669 / 1.553 / 2.019 | 3.311 / 6.106 / 7.994 |
| Dispatch | 12 / 32 / 64 | 0.167 / 0.310 / 0.435 | 0.202 / 0.544 / 1.112 |
| Clinical | 5 / 20 / 40 | 0.050 / 0.069 / 0.153 | 0.080 / 0.122 / 0.214 |
Propagation remains beneficial at the longer horizons shown. Increasing H also changes the observed transition count and return scale, so these are not fixed-data-budget comparisons.
To address the concern that high-dimensional regressors are weak controls, we added source-validation-tuned ten-dimensional regressors using coverage summaries and OPE estimates. InCOPE leads on the original GridWorld and Tumor tests, while the best compact controls lead on Dispatch and Clinical. Our analysis indicates that these controls exploit informative OPE estimates and stable summary-to-value relationships. We therefore also examined reduced behavior–target overlap in Dispatch and 29 non-oracle target policies in Clinical at n=72/96/120, without retraining. InCOPE achieves aggregate MAEs of 0.3540 versus 0.5356 and 0.0576 versus 0.0654, respectively, against the best compact regressor. These results support more robust transfer by InCOPE’s structured estimator in the reported supplementary settings, where changes in coverage, policy composition and sample size can weaken source-learned scalar corrections. Detailed settings and analysis are in our response to Reviewer C2Vv, Q4 (Table C4).
If accepted, we will add these results and discussion, including the no-occupancy alternative, to the paper.
References
[1] Yin and Wang. Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learning. AISTATS, 2020, Section 2.2 and Algorithm 1.
[2] Xie, Ma, and Wang. Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling. NeurIPS, 2019, Section 3.
Dear Reviewer ucm3,
As the deadline of discussion phase is approaching, we are eager to know if our reply has adequately resolved all of your concerns. Please let us know if there are any remaining issues that need to be addressed.
Thank you for your time and effort.
Thank you for your feedback.
Q1 / W1. Alternative uses of source-task information.
Our Bayesian model-based OPE baseline fits reward/transition priors to the same source logs, updates them with current-task data, and averages target-policy values over posterior MDP samples. This compares value prediction with model-prior learning for evaluation. Both InCOPE variants have lower MAE in all four domains (Reviewer ZyhJ, Q1, Table Z1).
Q2 / W3.1–W3.3. Prior mismatch and applicability.
Theoretical clarifications. Corollary 2 gives test optimality if pretraining/test conditional means agree. Retaining its test-context-distribution condition and assuming Etest[J2+(fpre⋆(Cn))2]<∞, without equality we have
Etest[(fpre⋆(Cn)−J)2]=VEtest[Vartest(J∣Cn)]+BEtest[(fpre⋆(Cn)−Etest[J∣Cn])2].
V is the minimum test MSE from Cn; B penalizes conditional-mean mismatch.
Proof. Add/subtract Etest[J∣Cn] and expand. The cross term is zero: the predictor depends only on Cn; Etest[J−Etest[J∣Cn]∣Cn]=0.
If B≤ηV, η≥0, test MSE is at most (1+η)V: small mismatch keeps the ideal predictor near test-optimal. Empirically, trained models can retain advantages under shifts below; the frozen unified model beats evaluated conventional estimators on unseen SysAdmin at n=4/8 (Reviewer ZyhJ, Q2, Table Z2).
Protocol. We shifted rewards, transitions or policies of 20 new synthetic MDPs sampled from the Clinical source generator without retraining; we crossed H={5,10,20}, n={4,8,24,48} and four behavior–target preference pairings. MAE averages these conditions and, for InCOPE, five training seeds. Best OPE is the lowest aggregate MAE among WPDIS, time-pooled FQE, pooled-smoothed TMIS, DR-PPI and MB-OPE.
Reward scaling. We scaled rewards and held transitions/policies fixed.
Table U1. Reward-scaling sensitivity (MAE).
Divide MAE by scale for cross-scale comparisons: specialist MAE is 0.1359 at 50%, 0.1184 at 175%, versus 0.0867 originally. Accuracy degrades on both sides; both models lead at 50%–175%, but WPDIS wins at 25%.
Transition changes. We mixed each nonterminal row with uniform mass on its original support or mass on its most probable successor(s); percentages are weights. We held rewards/policies fixed.
Table U2. Transition-shift sensitivity (MAE).
Errors grow away from original transitions. Both models beat Best OPE at 75% concentration but lose at 100%, where the most probable successor is always the current state: nonterminal self-loops change return accumulation despite fixed rewards. Averaging over H, this holds at each tested n∈{4,8,24,48}; no crossover shift is observed on this grid.
Policy changes. Uniform is uniform over actions; concentrated retains the most probable action(s). We varied both jointly and held the MDP fixed.
Table U3. Behavior–target policy combinations (MAE).
Both models lead in aggregate MAE in all nine combinations. Target concentration increases error; behavior-policy effects are smaller and not uniformly monotonic.
Shifts can reduce accuracy before removing relative advantage. If accepted, we will add these findings.