acceptodds
Under review as a conference paper at ICLR 2027

Prior-Data Fitted Networks for In-Context Off-Policy Evaluation

Abstract

Off-policy evaluation (OPE) is essential for assessing decision policies before deployment when online experimentation is costly or unsafe. Accurate estimation remains difficult when behavior-policy data are limited and the target policy favors states and actions rarely observed in those data. We propose InCOPE, a prior-data fitted network that learns a shared policy-value prediction rule across synthetic OPE tasks. We construct an OPE task prior over Markov decision processes and behavior–target policy pairs, generating behavior-policy datasets and exact target-policy values for pretraining. For a new tabular decision problem, the pretrained network uses the behavior-policy dataset, target policy, and known task settings as context to predict the policy’s expected return without task-specific parameter updates. Experiments across four tabular domains, including a semi-synthetic clinical setting, demonstrate InCOPE's effectiveness for policy-value estimation.

Primary area:
reinforcement learning
Keywords:
Off-policy evaluation, Prior-data fitted networks, In-context learning, Cross-task generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ucm3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • Proposes InCOPE, a prior-data fitted network for off-policy evaluation (OPE), pretrained on synthetic MDPs with exact target-policy values obtained via dynamic programming.

  • Uses behavior-policy trajectories, target-policy information, and estimated transition dynamics to predict policy values without task-specific parameter updates.

  • Evaluates the method across four tabular domains, reporting improvements over conventional OPE estimators and conducting ablations on pretraining priors and current-task information.

Critical strengths
  • Meaningful problem: Leveraging cross-task knowledge to improve OPE under limited data and substantial policy mismatch is a valuable research direction.

  • Promising empirical results: InCOPE consistently achieves lower MAE than the evaluated baselines across four domains. The paper is clearly presented, with detailed methodology and useful ablations.

Critical weaknesses

W1. The experiments do not sufficiently establish the benefit of the proposed estimator beyond the use of additional cross-task prior information. InCOPE is pretrained on synthetic MDPs with exact policy-value labels, whereas the six main OPE baselines only use current-task trajectories. This comparison is reasonable for demonstrating the benefits of pretraining, but cannot isolate whether the improvements arise from the proposed direct policy-value prediction mechanism or simply from access to additional task information. Since the source tasks are generated from known MDPs, the same source information could be used to learn transferable reward/transition models or Bayesian priors, followed by model-based evaluation using current-task observations. Approaches with clearer and more careful ablations would provide a more direct comparison of how prior information should be incorporated into OPE.

W2. Existing controls are insufficient: Section 5.5 compares InCOPE against Ridge, kNN, RBF-SVR, and ExtraTrees using complete task inputs. However, these regressors operate on a 98,228-dimensional flattened representation with only 3,000 source-training queries per domain (Appendix C.6). They do not provide a strong alternative that explicitly exploits the MDP structure or Bellman equations. Consequently, the results establish the usefulness of the overall pretrained framework more convincingly than the distinctive advantage of InCOPE's estimator design.

W3. The robustness and transferability of the synthetic prior are insufficiently characterized, limiting the practical significance of the approach.

  • W3.1. Practical implications of theoretical assumptions: Proposition 1 characterizes the Bayes-optimal predictor under the pretraining prior, while Corollary 2 establishes optimality under conditional-mean compatibility between training and test distributions. Since exact compatibility is unlikely in practice, it would be more valuable to understand how violations of this assumption affect estimation accuracy and real-world applicability.

  • W3.2. Insufficient evaluation of prior mismatch: Section 5.4 modifies source reward distributions and policy ranges, demonstrating that the pretraining prior matters. However, it does not systematically examine performance degradation as test-time reward mechanisms, transition dynamics, or policy-generation mechanisms deviate from the source prior.

  • W3.3. Unclear conditions for successful transfer: The paper does not establish which aspects of task similarity matter most, how prior mismatch interacts with data availability or coverage, or how much mismatch InCOPE can tolerate before its advantage disappears or negative transfer occurs. These findings would provide important practical guidance for deploying pretrained OPE estimators.

Questions for authors

Q1. Alternative use of source-task information.

Could the authors compare InCOPE against a prior-aware model-based OPE approach trained using the same source MDPs and supervision? For example, learning transferable reward/transition priors followed by current-task model estimation and dynamic programming would help distinguish the benefit of cross-task knowledge from that of direct policy-value prediction.

Q2. Sensitivity to prior mismatch.

Could the authors provide controlled experiments that progressively vary the similarity between source and test MDPs, separately considering reward functions, transition dynamics, and policy-generation mechanisms?

In particular, at what degree of mismatch does InCOPE lose its advantage over current-task-only OPE estimators, and how does this threshold depend on the number of available trajectories? Such results would make the method's applicability much clearer.

Q3. Oracle-derived target policies.

In Appendix C.1, target policies in several domains are constructed from true-MDP action values, such as (Q_0^u). Although these values are not directly provided to InCOPE, the resulting policy probabilities can encode information about the underlying environment.

Would the reported improvements persist when evaluating target policies generated independently of the true MDP parameters, such as randomly generated policies or policies learned exclusively from offline data? This would help determine the extent to which performance depends on the benchmark's policy-construction procedure.

Q4. Long-horizon value modeling.

Section 4.2 uses only two learned propagation blocks, while Appendix B.1 already computes empirical target-policy occupancy statistics through multi-step transition propagation.

Could the authors disentangle the contribution of these precomputed occupancy features from learned propagation, and evaluate how performance changes with horizon and propagation depth? This would clarify whether the model learns transferable long-horizon value relationships or primarily exploits information already available in the engineered inputs.

5 comments

Thank you for your feedback.

Q1 / W1. Alternative uses of source-task information.

Our Bayesian model-based OPE baseline fits reward/transition priors to the same source logs, updates them with current-task data, and averages target-policy values over posterior MDP samples. This compares value prediction with model-prior learning for evaluation. Both InCOPE variants have lower MAE in all four domains (Reviewer ZyhJ, Q1, Table Z1).

Q2 / W3.1–W3.3. Prior mismatch and applicability.

Theoretical clarifications. Corollary 2 gives test optimality if pretraining/test conditional means agree. Retaining its test-context-distribution condition and assuming Etest[J2+(fpre⋆(Cn))2]<∞\mathbb E_{\rm test}[J^2+(f_{\rm pre}^\star(C_n))^2]<\infty, without equality we have

Etest[(fpre⋆(Cn)−J)2]=Etest[Var⁡test(J∣Cn)]⏟V+Etest[(fpre⋆(Cn)−Etest[J∣Cn])2]⏟B.\mathbb E_{\rm test}[(f_{\rm pre}^\star(C_n)-J)^2]=\underbrace{\mathbb E_{\rm test}[\operatorname{Var}_{\rm test}(J\mid C_n)]}_{V}+\underbrace{\mathbb E_{\rm test}[(f_{\rm pre}^\star(C_n)-\mathbb E_{\rm test}[J\mid C_n])^2]}_{B}.

VV is the minimum test MSE from CnC_n; BB penalizes conditional-mean mismatch.

Proof. Add/subtract Etest[J∣Cn]\mathbb E_{\rm test}[J\mid C_n] and expand. The cross term is zero: the predictor depends only on CnC_n; Etest[J−Etest[J∣Cn]∣Cn]=0\mathbb E_{\rm test}[J-\mathbb E_{\rm test}[J\mid C_n]\mid C_n]=0.

If B≤ηVB\leq\eta V, η≥0\eta\geq0, test MSE is at most (1+η)V(1+\eta)V: small mismatch keeps the ideal predictor near test-optimal. Empirically, trained models can retain advantages under shifts below; the frozen unified model beats evaluated conventional estimators on unseen SysAdmin at n=4/8 (Reviewer ZyhJ, Q2, Table Z2).

Protocol. We shifted rewards, transitions or policies of 20 new synthetic MDPs sampled from the Clinical source generator without retraining; we crossed H={5,10,20}, n={4,8,24,48} and four behavior–target preference pairings. MAE averages these conditions and, for InCOPE, five training seeds. Best OPE is the lowest aggregate MAE among WPDIS, time-pooled FQE, pooled-smoothed TMIS, DR-PPI and MB-OPE.

Reward scaling. We scaled rewards and held transitions/policies fixed.

Table U1. Reward-scaling sensitivity (MAE).

ScaleSpecialistUnifiedBest OPE
25%0.07380.08750.0713
50%0.06800.07830.1426
75%0.07190.07610.2140
100%0.08670.09050.2853
125%0.11970.12080.3566
150%0.16220.15850.4279
175%0.20710.19900.4993

Divide MAE by scale for cross-scale comparisons: specialist MAE is 0.1359 at 50%, 0.1184 at 175%, versus 0.0867 originally. Accuracy degrades on both sides; both models lead at 50%–175%, but WPDIS wins at 25%.

Transition changes. We mixed each nonterminal row with uniform mass on its original support or mass on its most probable successor(s); percentages are weights. We held rewards/policies fixed.

Table U2. Transition-shift sensitivity (MAE).

Transition changeSpecialistUnifiedBest OPE
Uniformize 100%0.22260.20850.3902
Uniformize 75%0.17180.16260.3606
Uniformize 50%0.12920.12340.3432
Uniformize 25%0.09720.09750.3149
Original0.08670.09050.2853
Concentrate 25%0.12670.12080.3187
Concentrate 50%0.22660.20950.3944
Concentrate 75%0.38100.35140.4232
Concentrate 100%0.62590.58280.4593

Errors grow away from original transitions. Both models beat Best OPE at 75% concentration but lose at 100%, where the most probable successor is always the current state: nonterminal self-loops change return accumulation despite fixed rewards. Averaging over H, this holds at each tested n∈{4,8,24,48}; no crossover shift is observed on this grid.

Policy changes. Uniform is uniform over actions; concentrated retains the most probable action(s). We varied both jointly and held the MDP fixed.

Table U3. Behavior–target policy combinations (MAE).

BehaviorTargetSpecialistUnifiedBest OPE
UniformUniform0.08360.09430.2330
UniformOriginal0.08450.08710.2898
UniformConcentrated0.12770.17410.3810
OriginalUniform0.08680.09740.2489
OriginalOriginal0.08670.09050.2853
OriginalConcentrated0.12590.17860.3548
ConcentratedUniform0.09660.11190.3107
ConcentratedOriginal0.09800.09940.3495
ConcentratedConcentrated0.13230.19700.3492

Both models lead in aggregate MAE in all nine combinations. Target concentration increases error; behavior-policy effects are smaller and not uniformly monotonic.

Shifts can reduce accuracy before removing relative advantage. If accepted, we will add these findings.

Q3. Target policies constructed without oracle information.

To address this concern, we evaluated the frozen Clinical specialist on 29 target policies constructed without using the true transition, reward or Q functions: one uniform policy, 25 fixed action-biased policies (90% on a designated action plus 10% uniform exploration), and three state-dependent random policies. The Clinical MDPs were unchanged, and InCOPE was not retrained.

Averaging over these policies, H=5/10/20, n=72/96/120 and five training seeds, InCOPE achieves MAE 0.0576, compared with 0.0654 for the best of the four source-validation-tuned compact regressors. Thus, the advantage in these Clinical settings is also observed when target policies are generated independently of the true MDP parameters. The detailed setup and comparison are in our response to Reviewer C2Vv, Q4 (Table C4). If accepted, we will include this evaluation in the paper.

Q4 / W2. Occupancy features, learned propagation and compact controls.

Removing learned propagation increases mean MAE in all four original test domains, even with occupancy inputs retained; the best depth among L=1,2,4 varies by domain. Our occupancy summaries are auxiliary inputs motivated by established OPE approaches that estimate target-policy visitation from offline data [1,2]. Their removal improves Dispatch but increases mean error in the other three domains. These findings support treating them as optional inputs, with the no-occupancy model as an alternative configuration. The detailed ablation results are provided in our response to Reviewer C2Vv, Q4 (Table C3).

We further examined horizon dependence using 50 paired evaluation queries per domain, fixing the MDP, policies and trajectory count and using nested trajectory prefixes. Below are selected horizon settings, including extensions beyond the source-training range. Values are specialist MAEs averaged over five training seeds, in the listed H order.

Table U4. Horizon and propagation ablations (MAE).

DomainHFull (L=2)No propagation (L=0)
GridWorld12 / 24 / 480.300 / 0.897 / 1.4160.348 / 1.100 / 1.715
Tumor25 / 50 / 1000.669 / 1.553 / 2.0193.311 / 6.106 / 7.994
Dispatch12 / 32 / 640.167 / 0.310 / 0.4350.202 / 0.544 / 1.112
Clinical5 / 20 / 400.050 / 0.069 / 0.1530.080 / 0.122 / 0.214

Propagation remains beneficial at the longer horizons shown. Increasing H also changes the observed transition count and return scale, so these are not fixed-data-budget comparisons.

To address the concern that high-dimensional regressors are weak controls, we added source-validation-tuned ten-dimensional regressors using coverage summaries and OPE estimates. InCOPE leads on the original GridWorld and Tumor tests, while the best compact controls lead on Dispatch and Clinical. Our analysis indicates that these controls exploit informative OPE estimates and stable summary-to-value relationships. We therefore also examined reduced behavior–target overlap in Dispatch and 29 non-oracle target policies in Clinical at n=72/96/120, without retraining. InCOPE achieves aggregate MAEs of 0.3540 versus 0.5356 and 0.0576 versus 0.0654, respectively, against the best compact regressor. These results support more robust transfer by InCOPE’s structured estimator in the reported supplementary settings, where changes in coverage, policy composition and sample size can weaken source-learned scalar corrections. Detailed settings and analysis are in our response to Reviewer C2Vv, Q4 (Table C4).

If accepted, we will add these results and discussion, including the no-occupancy alternative, to the paper.

References

[1] Yin and Wang. Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learning. AISTATS, 2020, Section 2.2 and Algorithm 1.

[2] Xie, Ma, and Wang. Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling. NeurIPS, 2019, Section 3.

Dear Reviewer ucm3,

As the deadline of discussion phase is approaching, we are eager to know if our reply has adequately resolved all of your concerns. Please let us know if there are any remaining issues that need to be addressed.

Thank you for your time and effort.

Reviewer_ZyhJOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Focusing on tabular RL problems, we are presented with a pretrained transformer that estimates the value of a target policy from a small dataset collected under a different behaviour policy, without any training on the new task. This is a good idea, as exact labels are cheap in tabular MDPs, and small-sample OPE is where prior knowledge can help the most. The papers ports/translates the prior-data-fitted network idea (TabPFN, Hollmann and colleagues, 2022-2025) to off-policy evaluation in finite tabular MDPs. The dataset here is a behaviour-policy log plus a target policy, and the label is the target policy’s true value.

Critical strengths

A clean, principled transfer of PFNs to a problem where they fit quite naturally - in fact, the idea is so neat that most readers will think about why they did not think about it earlier this way.

Critical weaknesses

A natural baseline is missing. If the network approximates a posterior mean, the obvious comparison is an explicit Bayesian estimator. That would show whether the gains come from the learned domain prior or simply from shrinkage that any sensible prior provides.

Questions for authors

Can we get a natural baseline comparison? e.g. could you compare against a Bayesian model-based estimator that integrates over posterior uncertainty: independent Dirichlet priors on transition rows and Normal/Normal–Gamma priors on rewards, with value averaged over posterior samples. Fit the prior hyperparameters by empirical Bayes on the same source tasks InCOPE is pretrained on (fitting the prior on the source tasks matters for fairness. InCOPE has seen thousands of related tasks, and a Bayesian baseline should be allowed to learn its prior concentration from them too).

Curiosity question for the discussion to explore: Does the learned prior operate safely outside its training distribution? And how does it fail?

1 comment

Thank you for your positive assessment and helpful suggestions.

Q1. Source-informed Bayesian comparison.

Adapting Bayesian model-based OPE [1], we fitted independent Dirichlet transition priors and Normal reward-mean priors to the same source-task logs used for the corresponding InCOPE specialist. Model selection uses source-validation data. At test time, current-task logs update the posterior; target-policy values are averaged over 4,096 posterior transition samples, with reward means integrated analytically.

Results on the complete original test sets are reported as MAE ± SD across five test blocks.

Table Z1. Source-fitted Bayesian comparison (MAE ± SD).

MethodGridWorldTumorDispatchClinical
Bayesian method1.0346 ± 0.09782.9552 ± 0.12121.0343 ± 0.02030.5405 ± 0.0105
InCOPE specialist0.7654 ± 0.06460.6818 ± 0.01270.3179 ± 0.00370.0655 ± 0.0048
InCOPE unified0.7659 ± 0.05910.5103 ± 0.01300.4491 ± 0.00680.0870 ± 0.0063

Both InCOPE variants outperform this source-fitted Bayesian baseline in all four domains. If accepted, we will include this comparison and implementation details in the final paper.

Q2. Transfer to an unseen task family.

SysAdmin [2] models a computer network where machine failures depend on neighboring machines, and reboot actions trade off uptime against cost. We use small-network instances of its IPPC2011 implementation [3]: 3- and 4-machine networks with horizons of 24 and 40, respectively, and a discount factor of 0.97. The unified model pretrained on our four task families is evaluated directly, without SysAdmin-specific training.

The table reports mean MAE ± approximate 95% t-interval half-widths across five test blocks, each containing six sampled networks and two target policies per network. InCOPE is averaged over five pretrained models; n denotes the number of behavior-policy trajectories. FQE pools transitions across time; Tabular-MIS uses pooled transitions with a fixed total pseudocount of 0.5 per state-action row. OPERA uses these updated estimators and numerically stabilized importance weights.

Table Z2. Transfer to SysAdmin (MAE ± approximate 95% CI half-width).

Methodn=4n=8n=12
WPDIS2.7237 ± 1.03632.4519 ± 1.16302.4286 ± 0.5957
FQE4.2887 ± 1.76771.8471 ± 0.72300.7064 ± 0.2382
Tabular-MIS2.5720 ± 0.74181.2373 ± 0.21220.7298 ± 0.2748
OPERA10.4851 ± 2.77697.5508 ± 2.14005.0208 ± 1.0766
DR-PPI14.4893 ± 0.566114.1140 ± 0.531013.8103 ± 0.4883
MB-OPE2.5720 ± 0.74181.2373 ± 0.21220.7298 ± 0.2748
InCOPE unified0.8102 ± 0.05880.8030 ± 0.11110.7841 ± 0.0441

InCOPE leads at n=4 and n=8, showing effective transfer with very limited data. FQE achieves the lowest MAE at n=12, which is expected, as estimators based on observations from the current task generally improve with increasing sample size. If accepted, we will add these results and discussion to the final paper.

References

[1] Sonabend-W et al. Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation. NeurIPS, 2020.

[2] Guestrin, Koller, and Parr. Max-norm Projections for Factored MDPs. IJCAI, 2001.

[3] https://github.com/pyrddlgym-project/rddlrepository/tree/3412a7b62d403f51ed737960656890cae73a369e/rddlrepository/archive/competitions/IPPC2011/SysAdmin/MDP

Reviewer_C2VvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper applies PFN to off-policy evaluation in finite-horizon tabular MDPs. A neural network is pretrained on synthetic OPE tasks with exact policy-value labels computed by dynamic programming. At test time, it predicts target-policy returns from behavior-policy statistics, target-policy probabilities, and known task settings without additional training. Across GridWorld, Tumor, Dispatch, and semi-synthetic ICU-Sepsis, InCOPE achieves lower MAE than six conventional OPE estimators.

Critical strengths
  1. The paper explores a promising direction by using knowledge learned across related tasks to improve OPE when historical data are limited. The experimental protocol is carefully specified, with exact value labels, disjoint training and validation MDPs, and informative ablations. The unified model outperforms all six OPE baselines across four domains without requiring domain identifiers.
Critical weaknesses
  1. The evaluation provides limited evidence of generalization beyond the designed task priors. Although source and test generators differ, they are developed within the same task families. The Clinical evaluation also uses aggregations of a single published ICU-Sepsis MDP. Table 2 shows substantial sensitivity to prior design, yet no independently developed OPE environment is evaluated. It remains unclear how reliably InCOPE transfers beyond the task families used to design its priors, limiting evidence for broader applicability.

  2. The evaluation does not isolate the benefit of the proposed architecture from prior knowledge and baseline choices. None of the six OPE baselines uses the source-task supervision available to InCOPE. The log-free model already achieves MAEs numerically below the best conventional baselines in three domains, although on separate test sets. No prior-informed Bayesian OPE baseline is included. Furthermore, FQE and Tabular-MIS use time-specific estimates without smoothing despite time-homogeneous dynamics. The contributions of pretraining, architecture, and baseline implementation remain difficult to distinguish, leaving the proposed architecture's added value insufficiently established.

Questions for authors
  1. Table 2 shows that narrowing the source-policy range increases GridWorld MAE from 0.769 to 0.995, while replacing the original source rewards raises MAE from 0.764 to 1.679 on GridWorld and from 0.660 to 3.829 on Tumor. Can the authors evaluate InCOPE on independently developed OPE environments or with learned target policies? Could predictive uncertainty be estimated and validated under prior mismatch?

  2. All Clinical test blocks derive from one ICU-Sepsis MDP, using different aggregations to 48 live states. What environments were used for the separate development evaluations mentioned in Appendix C.1? How would InCOPE perform on the original 716-state MDP or with behavior policies estimated from clinicians' decisions?

  3. On Dispatch, FQE, Tabular-MIS, and OPERA report MAEs around 13 (Table 1), compared with 1.897 for a constant source-mean predictor on a separate test set (Table 16). Why do these estimators perform so poorly, and would this gap persist in a matched comparison with time-pooled, smoothed baselines? How does InCOPE compare with Bayesian model-based OPE using a prior fitted to the same source tasks?

  4. Can the authors compare a compact regressor trained on the same source queries, using conventional OPE estimates and coverage statistics as inputs? The full-input regression controls in Table 3(b) use 98,228-dimensional inputs with untuned settings and only 3,000 training queries. Can the estimated target-occupancy features and policy-conditioned propagation blocks also be ablated separately?

5 comments

Thank you for the careful review.

Q1. Generalization and uncertainty under prior mismatch.

The frozen unified model outperforms all six conventional estimators on unseen SysAdmin (computer-network maintenance) with four or eight trajectories (ZyhJ Q2, Table Z2). InCOPE is a point estimator; the approximate 95% CIs concern mean MAE across five test blocks. Controlled reward, dynamics and policy changes show that pretraining helps without an exact training-distribution match, but large reward-scale or dynamics shifts can let conventional OPE using only current-task data perform better (ucm3 Q2, Tables U1–U3).

Q2. Development/test separation and additional Clinical evaluations.

Appendix C.1 development evaluations guided synthetic training-distribution design, as development data guide TabPFN prior design [1,2]. GridWorld, Tumor and Dispatch development/test MDPs share simulation rules, with separate draws of, respectively: map layouts and transition/reward parameters; tumor-growth, treatment-effect and noise parameters; demand, movement and cost parameters. Clinical uses five development and five different test clusterings of the released 716-state ICU-Sepsis model. Each groups 713 nonterminal states into 48 clusters and keeps three terminals (51 states), yielding different transitions, expected rewards and initial-state distributions. Each MDP has separately generated policies and trajectories.

We also tested the frozen models with (i) the released clinician-derived behavior policy in 51 states and (ii) synthetic behavior policies in the unaggregated 716-state MDP. Each evaluation has 1,200 queries, H=5/10/20 and n=24/48/96/160 trajectories. FQE pools transitions across time; Tabular-MIS additionally smooths pooled transitions, with smoothing strength fixed on original source-validation data.

Table C1. Additional Clinical evaluations (MAE).

Method51 states, clinician behavior716 states, synthetic behavior
InCOPE specialist0.06470.0995
InCOPE unified0.10910.1688
WPDIS0.25200.2404
FQE0.41950.4903
Tabular-MIS0.25510.4480
DR-PPI0.36530.4152
MB-OPE0.25990.4542

Both frozen InCOPE variants have lower aggregate MAE than all five conventional estimators in both settings. If accepted, we will clarify development/test protocols and add these results.

Q3. Time-pooled baselines and Bayesian OPE.

Original FQE and Tabular-MIS estimated each time step separately, leaving sparse state-action counts even with informative pooled data. In Dispatch, zeros for unobserved cells contribute to underestimation, also affecting OPERA through its constituents.

We reran all four original test sets with time-pooled FQE [3] and a time-homogeneous Tabular-MIS adaptation pooling transitions and rewards [4], with optional Dirichlet transition pseudocounts [5]. Source validation selected parameters, allowing zero smoothing. We recomputed OPERA with updated estimators and numerically stabilized importance weights. All FQE/TMIS results in our responses, including compact-regression inputs, use these versions.

Table C2. Updated baselines on the four original test sets (MAE).

MethodGridWorldTumorDispatchClinical
FQE, time-pooled0.91140.73514.96410.3659
TMIS, time-pooled0.91151.50450.97170.2063
OPERA, updated1.00421.33443.12320.4910
Bayesian method1.03462.95521.03430.5405
InCOPE specialist0.76540.68180.31790.0655
InCOPE unified0.76590.51030.44910.0870

FQE/TMIS errors fall substantially, but both InCOPE variants retain lower aggregate MAE in every domain. The Bayesian method learns reward and transition priors from the same source-task logs (implementation and five-block variability: ZyhJ Q1, Table Z1). If accepted, we will update baseline results and implementation details.

References

[1] Hollmann et al. TabPFN. ICLR 2023.

[2] Hollmann et al. Accurate predictions on small data with a tabular foundation model. Nature 2025.

[3] Zhang et al. Off-Policy Fitted Q-Evaluation. ICML 2022, §4.1.

[4] Ren et al. Nearly Horizon-Free Offline RL. NeurIPS 2021, §4.1.

[5] Rathnam et al. The Unintended Consequences of Discount Regularization. ICML 2023, §4.

Q4. Component ablations and compact regression controls.

Component ablations. We retrained on the same source data and budgets, selecting on source validation. Table C3 reports mean MAE over five training replicates on the original test sets.

Table C3. Component and propagation-depth ablations (MAE).

ConfigurationGridWorldTumorDispatchClinical
Full, L=20.76540.68180.31790.0655
Without occupancy summaries0.83600.70290.27210.0725
Without propagation, L=00.88393.33630.53170.1053
L=10.78570.83750.30610.0679
L=40.75460.91240.29180.0761

Removing learned propagation raises mean error in every domain; the best nonzero depth varies. Occupancy summaries draw on OPE methods estimating target-policy visitation from offline transitions [6,7]. Removing them improves Dispatch but worsens the other three domains. We view them as optional inputs and the no-occupancy model as an alternative configuration; if accepted, we will include it and discuss when summaries help.

Compact regression controls. These ten-dimensional controls are substantially more competitive than flattened-input regressors. Inputs are log trajectory count, log horizon, discount, two target-coverage/count summaries, and WPDIS, FQE, TMIS, DR-PPI and MB-OPE estimates.

Each replicate uses 3,000 source-training and 600 separately generated source-validation queries. Inputs/labels are standardized on training data only. Validation MAE selects parameters below, then all models are frozen:

  • Ridge: regularization {0.001, 0.01, 0.1, 1, 10, 100, 1000}.
  • kNN: {3, 5, 10, 20, 50} neighbors; equal/inverse-distance weights.
  • RBF-SVR: error-penalty C={0.1, 1, 10, 100}; kernel gamma (distance decay)=0.1 or 1/(10 × training-input variance); unpenalized-error epsilon={0.05, 0.2} in standardized label units.
  • ExtraTrees: min. leaf size {1, 3, 10, 30}; features per split {50%, 100%}; 256 trees, unlimited depth.

“Best compact” is the lowest aggregate MAE among four fixed regressors, not per-query selection. Supplementary rows equally average listed n values; Clinical also averages H=5/10/20.

Table C4. Compact regression comparisons (MAE).

Evaluation settingInCOPE specialistBest compact
GridWorld: original full test0.76540.8273
Tumor: original full test0.68181.4509
Dispatch: original full test0.31790.2805
Clinical: original full test0.06550.0570
Dispatch: behavior mismatch, H=32, n=4/8/240.35400.5356
Clinical: 29 target policies, n=72/96/1200.05760.0654

Compact regressors learn a source-task correction from OPE estimates and coverage/horizon summaries. In original Dispatch, target-relevant coverage is high at larger n and MB-OPE becomes accurate, providing informative correction inputs. In Clinical, values cluster by horizon, making global summaries predictive. These advantages do not extend equally to GridWorld/Tumor, where compact controls remain less accurate than InCOPE.

Both supplementary evaluations keep source data and fitted models unchanged. Dispatch retains 50 original H=32 MDPs and target policies but mixes the original behavior policy 50:50 with one favoring target-low-probability actions, testing correction under weaker behavior-target overlap. Clinical retains five original MDPs and broadens to 29 non-oracle target policies: one uniform, 25 fixed action-biased, three state-dependent random. Varying n also tests correction as OPE estimates and coverage change.

InCOPE has lower aggregate MAE than all four regressors in both supplementary evaluations (Table C4). In Clinical, its MAE falls from 0.0720 to 0.0563 as n rises from 24 to 120, leading at n=72,96,120. Compact regression benefits from informative OPE estimates and stable summary-to-value relationships, but coverage, policy-composition or sample-size changes can disrupt its correction. InCOPE retains low aggregate error across these reported conditions; with the propagation ablation, this supports more robust transfer of its structured estimator than compact scalar correction in these settings. If accepted, we will add results, full sweeps and discussion.

References

[6] Yin and Wang. Asymptotically Efficient OPE for Tabular RL. AISTATS 2020, §2.2, Alg. 1.

[7] Xie, Ma, Wang. Towards Optimal OPE with Marginalized IS. NeurIPS 2019, §3.

Dear Reviewer C2Vv,

As the deadline of discussion phase is approaching, we are eager to know if our reply has adequately resolved all of your concerns. Please let us know if there are any remaining issues that need to be addressed.

Thank you for your time and effort.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of responses

Dear Area Chair,

We sincerely thank you and the reviewers for your time, careful reading, and constructive feedback. To assist with your assessment, we summarize our main responses below and point to the relevant results in the discussion.

Reviewer C2Vv noted that the additional results and clarifications helped address their concerns about generalization and baseline comparisons, and indicated that they would update their rating. In response to Reviewer ucm3’s follow-up, we reported two further comparisons: a Bayesian baseline with access to the exact source MDPs, and matched tests showing when InCOPE outperforms compact regressors.

We have organized the evidence from our initial rebuttal and follow-up around four main questions raised in the reviews:

  1. Does the advantage persist against stronger baselines? We reran time-pooled FQE and Tabular-MIS and recomputed OPERA. We also fitted the Bayesian baseline using the exact transition probabilities and expected rewards from the same source MDPs used by InCOPE. Both InCOPE variants retained lower MAE than these baselines on all four original benchmarks. See C2Vv (Table C2), ZyhJ (Table Z1), and our reply to ucm3’s follow-up (Q1).

  2. When does the structured estimator help? Tuned compact regressors lead on the original Dispatch and Clinical benchmarks, while InCOPE leads on GridWorld and Tumor. Among the log sizes reported for Dispatch, the best compact regressor overtakes InCOPE at 24 trajectories under high behavior–target overlap, but not until 128 under the lowest overlap tested. In Clinical, changing only the reward weakens the original horizon-to-value relationship, and InCOPE leads the best compact regressor at three of the four nonzero penalty weights. Separately, removing learned propagation increases mean MAE in all four domains, even with occupancy summaries retained. See C2Vv (Tables C3–C4) and our reply to ucm3’s follow-up (Q2, Tables 1–2).

  3. How far does the learned prior transfer? Without further training, the unified model achieves the lowest MAE on unseen SysAdmin with four or eight trajectories; time-pooled FQE leads with twelve. Controlled reward, transition, and policy changes in synthetic Clinical tasks show that InCOPE can retain its advantage under mismatch, but loses it under some large reward or transition shifts. See ZyhJ (Table Z2) and ucm3 (Tables U1–U3).

  4. Do the Clinical results depend on the original evaluation setup? We clarified that the development and test environments use different aggregations of the same released ICU-Sepsis MDP. Both InCOPE variants outperform the evaluated conventional estimators in two separate tests: one with a clinician-derived behavior policy and one on the unaggregated 716-state MDP. The specialist also achieves lower aggregate MAE than the best compact regressor in an evaluation with 29 target policies constructed without using the true MDP parameters. See C2Vv (Tables C1 and C4) and ucm3 (Q3).

If accepted, we will incorporate these results and clarify the evaluation protocols in the final paper. We are grateful to the reviewers for their thoughtful feedback and for engaging with our responses. Thank you for your time and for considering our submission.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.