acceptodds
Under review as a conference paper at ICLR 2027

A Foundation Model for Policy In-Context Longitudinal Causal Effect Estimation

Abstract

Longitudinal causal effect estimation is complicated by time-varying confounding and treatment-confounder feedback loops. Existing methods generally require data-specific nuisance-model fitting, and many are tailored to prespecified treatment sequences or prediction horizons, leading to time-consuming computation at deployment. We introduce *Policy In-Context Longitudinal causal Effect estimation with Prior-data Fitted Networks* (**PICLE-PFN**), the first foundation model for longitudinal causal effect estimation that supports *dynamic* treatment policies. PICLE-PFN represents target policies through in-context history-decision demonstrations, providing a unified interface for a broad class of treatment policies. Pretrained over synthetic data sampled from longitudinal structural causal models, PICLE-PFN amortizes causal effect estimation across datasets, policies, and horizons. Given a new observational dataset and target policy, it directly predicts conditional average potential outcomes (CAPOs) in one forward pass without task-specific training or hyperparameter tuning. Across synthetic and semi-synthetic longitudinal benchmarks, PICLE-PFN achieves comparable and lower RMSE and substantially improved uncertainty calibration, compared to state-of-the-art conventional estimators trained and tuned separately for each dataset.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Longitudinal Causal Inference, Prior-Data Fitted Networks, In-Context Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2ZnPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a foundation model for causal inference with longitudinal data, focusing on estimating conditional average potential outcomes under static and dynamic treatment policies. Its central identification strategy relies on sequential ignorability conditional on observed histories, together with consistency and positivity. The proposed approach combines a policy-demonstration representation with synthetic pretraining to enable estimation across datasets, policies, and horizons without task-specific retraining.

Critical strengths

Developing foundation models for causal inference is a valuable and promising research direction. The paper extends this approach to longitudinal settings, where evaluating multiple treatment policies and intervention horizons can be computationally demanding. The ability to reuse a pretrained model without fitting a separate model for each downstream task could be useful in practice, particularly for applications requiring repeated policy evaluation.

The proposed method offers an appealing connection between the statistical formulation of causal inference and machine learning. Representing a policy through its decisions on factual histories provides a unified way to supply policy information to the model. Proposition 3.1 gives a useful population-level justification for this representation: under the stated assumptions, the policy-demonstration distributions, together with the observational distribution, identify the target conditional average potential outcome. This connection strengthens the motivation for the proposed learning framework.

Critical weaknesses

Difference-in-differences (DiD) and synthetic control are among the most widely used approaches to causal inference with panel data. However, the proposed framework does not appear to encompass these approaches under their usual identifying assumptions; its identification argument instead relies on sequential ignorability. Consequently, the model’s generality appears to be primarily across datasets, policies, and horizons within a particular identification framework, rather than across panel-data causal inference more broadly. In this respect, the model is less “foundational” than I would expect from the paper’s framing. The authors should clarify the intended scope and discuss whether these important settings can be accommodated or remain outside the proposed framework.

The empirical evaluation is relatively narrow and does not fully substantiate the claimed ability to accommodate policies with different underlying functional forms. In particular, the main dynamic-policy benchmarks vary a scalar perturbation of the behavior-policy rule, rather than evaluating substantially different history-dependent decision rules. These experiments provide evidence of performance across related policy variants, but less evidence of generalization across qualitatively different policy families. A broader evaluation involving more diverse dynamic policies would make this central claim more convincing.

Questions for authors

Could the authors provide a clearer explanation of the uncertainty intervals and their intended statistical interpretation? In particular, do the intervals capture variability in individual counterfactual outcomes, uncertainty in estimating their conditional means, or both? The paper notes that these intervals should not be interpreted as credible intervals for CAPO, while evaluating their empirical coverage against the true CAPO. How should readers interpret the nominal interval levels and the resulting calibration analysis?

Would it be possible to develop a formal sensitivity analysis for violations of sequential ignorability? For example, could users specify the strength of unobserved confounding and assess how their estimated causal quantities or substantive conclusions would change? A discussion of how such an analysis could be incorporated into the proposed framework would help clarify its practical usefulness when the identifying assumption is questionable.

3 comments

W1 We thank the reviewer for the suggestion to clarify our scope. Indeed, our method aims to amortize CAPO estimation across datasets, policies, and horizons within the sequential backdoor adjustment framework, and does not intend to encompass all identification strategies for causal effect estimation on panel data. To make the scope explicit, we have revised the abstract (line 21) and introduction (line 74), in addition to the stated identifying assumption in Section 3.

DiD and synthetic control (SC) offer alternative identification strategies for panel data, and extensions such as Synthetic Blips [4] also accommodate sequential treatments. However, their identifying assumptions differ from those underlying PICLE-PFN. DiD relies on parallel-trends assumptions, potentially conditional on covariates [1], and the compatibility of parallel-trends assumptions with dynamic treatment policies requires careful consideration [2,3]. SC typically adopts latent-factor approaches to exploit additional structure across units and time [4–6]. These approaches can accommodate certain forms of unobserved confounding that are outside our sequential-ignorability framework. Extending PICLE-PFN to these settings would require establishing identification of the target estimands under the alternative assumptions and developing corresponding pretraining formulations and input representations. Such extensions remain outside the present study.

[1] Caetano, Carolina, et al. "Difference in differences with time-varying covariates." arXiv preprint arXiv:2202.02903 (2022).

[2] Stéphane Bonhomme, Back to feedback: dynamics and heterogeneity in panel data, The Econometrics Journal, 2026, utag027

[3] Marx, Philip, Elie Tamer, and Xun Tang. "Parallel trends and dynamic choices." Journal of Political Economy Microeconomics 2.1 (2024): 129-171.

[4] Agarwal, Anish, et al. "Synthetic Blips: Generalizing Synthetic Controls for Dynamic Treatment Effects." arXiv preprint arXiv:2210.11003 (2022).

[5] Arkhangelsky, Dmitry, and David Hirshberg. "Large-sample properties of the synthetic control method under selection on unobservables." arXiv preprint arXiv:2311.13575 (2023).

[6] Imbens, Guido W., and Davide Viviano. "Identification and inference for synthetic controls with confounding." arXiv preprint arXiv:2312.00955 (2023).

W2 We thank the reviewer for suggesting broader evaluations of policies with different underlying functional forms. We expand our evaluation to three dynamic policies that use qualitatively different history-dependent policy families, rather than scalar perturbations of the behavioral policy. These policies incorporate treatment history, outcome trends, and combinations of patient characteristics (definition details in Appendix D.8):

  • Deterioration: At step t, if the patient was not treated at t-1, start treatment if the target outcome increases relative to the previous step (deterioration); once treated, continue treatment.

  • Response-guided: At each step t, if the patient was not treated at t-1, start treatment when heart rate exceeds its 75th percentile; if the patient was treated at t-1, continue treatment while the target outcome is decreasing relative to the previous step, and stop it otherwise.

  • Tree: At each step t, treat if the target outcome increased relative to the previous step and the patient is older than the cohort median age, or if heart rate exceeds its 90th percentile; otherwise do not treat.

We compare PICLE-PFN with IGC-Net and D3, the two strongest baselines in our evaluation that support dynamic policies, across horizons of 2/4/6/8/10 with 20 repetitions. The table below summarizes RMSE averaged across horizons; detailed results for individual horizons are provided in Table 6.

policyPICLE-PFNIGC-NetD3
deterioration0.515 ± 0.1340.567 ± 0.1330.542 ± 0.127
response-guided0.748 ± 0.1730.783 ± 0.1820.779 ± 0.162
tree0.908 ± 0.2340.979 ± 0.2740.975 ± 0.258

PICLE-PFN achieves the lowest mean normalized RMSE at every evaluated horizon under both the deterioration and response-guided policies. Under the tree policy, the ranking varies across horizons, but PICLE-PFN achieves the lowest mean normalized RMSE aggregated across horizons. These results provide additional evidence that PICLE-PFN generalizes beyond scalar perturbations of the behavioral policy to diverse policies with different underlying functional forms.

Q1 We thank the reviewer for highlighting the need to clarify the statistical interpretation of our intervals. The original model’s predictive intervals capture both outcome variability and uncertainty about conditional means. Their coverage of true CAPO therefore does not directly establish calibration of CAPO uncertainty. We have revised the abstract, introduction, and results discussion to describe our findings as improved uncertainty characterization, supported by empirical CAPO coverage substantially closer to nominal levels than the baseline, rather than calibrated CAPO uncertainty.

To better align supervision with CAPO, we add 12 hours of second-stage pretraining after the initial 36 hours, replacing each single-draw target with the average of 10 independent counterfactual rollouts under the same sampled SCM and conditioning history. The two-stage procedure combines computationally efficient initial pretraining with subsequent training on targets more concentrated around CAPO. The resulting intervals approximate uncertainty about CAPO while retaining residual Monte Carlo variability from the finite number of rollouts.

We have updated the main benchmarks, coverage analysis, and ablations using this checkpoint, with comparisons to the original checkpoint in Appendix G. Point-estimation RMSE remains nearly unchanged (Figure S.4), while intervals become narrower (Figure S.5). At the nominal 95% level, empirical CAPO coverage is 88–93% on MIMIC-Y1 and 93–94% on MIMIC-1T across horizons, still substantially closer to nominal coverage than IGC-Net (Figure 4). These results support tighter intervals with strong empirical CAPO coverage and preserved point-estimation accuracy. This finding suggests that PICLE-PFN’s predictive intervals provide a useful characterization of uncertainty in CAPO estimation.

Q2 We thank the reviewer for this suggestion. A recent study [7] develops formal sensitivity bounds for average potential outcomes under violation of sequential ignorability, allowing users to specify the strength of unmeasured confounding (UC). These bounds can help assess whether substantive conclusions remain unchanged under the specified degree of confounding.

Incorporating this approach into PICLE-PFN is a promising direction. It would require estimating treatment propensity scores at each time step and implementing the associated sensitivity-bound optimization (Proposition 1 in [7]), beyond the outcome modeling provided by our current framework. Separate propensity models could supply the treatment probabilities, but adapting and validating the bound computation for our amortized CAPO estimation setting would require additional methodological development. We therefore leave this extension to future work.

As a complementary empirical assessment, we conduct two additional studies. The original submission considered a time-invariant UC under a static treatment sequence. Here, we evaluate a dynamic treatment policy, considering both time-invariant and time-varying UCs (Details see Appendix D.7). We compare PICLE-PFN with the two strongest baselines supporting dynamic treatment policies and report mean ± SD of RMSE across 20 repetitions.

(1) Time-invariant UC. U∼N(0,1)U \sim N(0,1), and the fixed UU enters the treatment logit as γU\gamma U and the outcome as 0.5U0.5U at every time step.

γ\gammaPICLE-PFNIGC-NetD3
00.363 ± 0.1000.372 ± 0.0970.381 ± 0.099
10.354 ± 0.0980.363 ± 0.0920.374 ± 0.096
20.355 ± 0.0920.362 ± 0.0910.380 ± 0.089
40.346 ± 0.0840.359 ± 0.0790.381 ± 0.080

(2) Time-varying UC. U1∼N(0,1),Ut=0.5Ut−1+0.75ϵt,ϵt∼N(0,1)U_1 \sim N(0,1), U_t = 0.5 U_{t-1} + \sqrt{0.75} \epsilon_t, \epsilon_t\sim N(0,1). The time-varying UtU_t enters the treatment logit as γUt\gamma U_t and the outcome as 0.5Ut0.5U_t at every time step.

γ\gammaPICLE-PFNIGC-NetD3
00.363 ± 0.1000.372 ± 0.0970.381 ± 0.099
10.488 ± 0.0600.485 ± 0.0640.527 ± 0.062
20.499 ± 0.0530.503 ± 0.0650.513 ± 0.049
40.524 ± 0.0470.528 ± 0.0680.531 ± 0.040

All methods exhibit relatively stable performance across confounding strengths in the time-invariant setting, whereas time-varying confounding increases RMSE for all methods. PICLE-PFN achieves the lowest mean RMSE in 7 out of 8 configurations, including all strong-confounding settings, maintaining its relative performance advantage under the evaluated violations of sequential ignorability.

[7] Tan, Zhiqiang. "Sensitivity models and bounds under sequential unmeasured confounding in longitudinal studies." Biometrika 112.1 (2025): asae044.

Thank you for the detailed clarifications and additional experiments. The revised scope and evaluations of qualitatively different dynamic policies substantially address my main concerns. I also appreciate the additional pretraining and the more cautious interpretation of uncertainty. The intervals still retain rollout variability, and reported CAPO coverage remains below nominal, so these limitations should remain explicit. The confounding experiments provide useful empirical assessments, with formal sensitivity analysis remaining an important extension. Overall, the response strengthens the paper, and I will maintain my rating of weak acceptance.

Reviewer_BLeEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PICLE-PFN, a foundation model for longitudinal causal effect estimation using policy in-context learning. The method represents static and dynamic treatment policies through history-decision demonstrations and employs a PFN-style Transformer pretrained on synthetic longitudinal structural causal models to directly estimate conditional average potential outcomes (CAPOs). Unlike conventional approaches, PICLE-PFN supports different datasets, policies, and prediction horizons without task-specific retraining. The authors provide a theoretical justification for the sufficiency of policy demonstrations under standard causal identification assumptions. Experiments on seven synthetic and semi-synthetic benchmark settings show that PICLE-PFN achieves the lowest RMSE in six settings, substantially reduces deployment time, and improves uncertainty calibration compared with existing methods. Additional experiments examine robustness to smaller sample sizes, unobserved confounding, and unseen policies, while a real-world MIMIC evaluation assesses factual outcome prediction.

Critical strengths

PICLE-PFN introduces a unified in-context learning framework that supports both static and dynamic treatment policies across different horizons without task-specific retraining. This addresses an important limitation of existing longitudinal causal estimators.

The method achieves the lowest RMSE in six of seven benchmark settings (Table 3), with up to 48.7% improvement over the strongest baseline. It also demonstrates substantially faster deployment (approximately 5000× compared with IGC-Net) and improved uncertainty calibration, supporting its practical value.

Critical weaknesses

In Section 5.2 (Table 4), the real-world MIMIC experiment evaluates only factual blood pressure prediction under observed treatments, rather than counterfactual outcomes under alternative policies. Although PICLE-PFN achieves slightly lower prediction errors than IGC-Net, this does not establish its ability to estimate causal effects in real clinical settings. The main causal estimation results therefore rely entirely on synthetic and semi-synthetic benchmarks.

PICLE-PFN is pretrained on synthetic longitudinal SCMs with specific structural mechanisms and policy families (Appendix B.2). However, the out-of-prior evaluation in Section 5.2 (Figure 5c) considers only two unseen static treatment sequences, (1,0,1,0) and (1,0,0,1), on a single benchmark with a four-step horizon. This limited experiment does not adequately demonstrate robustness to unseen dynamic policies or substantial shifts in causal mechanisms, which is particularly concerning for a foundation model intended to generalize across datasets and policies.

Section 4.2 trains the model using single-draw counterfactual outcomes rather than conditional mean targets. Consequently, the predictive distribution captures outcome variability in addition to estimation uncertainty. The authors explicitly acknowledge in Section 5.2 that the resulting intervals should not be interpreted as credible intervals for CAPO. Nevertheless, Figure 4 evaluates their coverage against true CAPO values and presents this as improved uncertainty calibration. This raises concerns about whether the reported calibration results meaningfully assess uncertainty in the estimated causal quantity.

Questions for authors

Please see my concerns in weakness.

3 comments

W1 We agree that factual prediction accuracy does not establish causal estimation accuracy in real clinical settings. Our causal estimation results rely on synthetic and semi-synthetic benchmarks, where counterfactual ground truth is available for direct evaluation. The real-world MIMIC experiment provides a sanity check using factual outcomes in real clinical data; it does not validate counterfactual predictions under alternative treatment policies. We have revised Section 5.2 (line 377, line 408-409) to make this distinction explicit.

W3 We thank the reviewer for highlighting this distinction. The original model’s predictive intervals capture both outcome variability and uncertainty about conditional means. Their coverage of true CAPO therefore does not directly establish calibration of CAPO uncertainty. We have revised the abstract, introduction, and results discussion to describe our findings as improved uncertainty characterization, supported by empirical CAPO coverage substantially closer to nominal levels than the baseline, rather than calibrated CAPO uncertainty.

To better align supervision with CAPO, we add 12 hours of second-stage pretraining after the initial 36 hours, replacing each single-draw target with the average of 10 independent counterfactual rollouts under the same sampled SCM and conditioning history. The two-stage procedure combines computationally efficient initial pretraining with subsequent training on targets more concentrated around CAPO. The resulting intervals approximate uncertainty about CAPO while retaining residual Monte Carlo variability from the finite number of rollouts.

We have updated the main benchmarks, coverage analysis, and ablations using this checkpoint, with comparisons to the original in Appendix G. Point-estimation RMSE remains nearly unchanged (Figure S.4), while intervals become narrower (Figure S.5). At the nominal 95% level, empirical CAPO coverage is 88–93% on MIMIC-Y1 and 93–94% on MIMIC-1T across horizons, still substantially closer to nominal coverage than IGC-Net (Figure 4). These results support tighter intervals with strong empirical CAPO coverage and preserved point-estimation accuracy. This finding suggests that PICLE-PFN’s predictive intervals provide a useful characterization of uncertainty in CAPO estimation.

W2 We thank the reviewer for the suggestion to broaden the evaluation regarding dynamic policy forms and causal mechanisms. Below, we address these two aspects separately.

  1. Generalization to unseen dynamic policies. Our original submission evaluated two static alternating sequences, which strictly lie outside our prior for static policies. We expand our evaluation to three dynamic policies whose explicit rule-based constructions are not used during pretraining, although they may not lie strictly outside our prior because the randomly weighted MLPs in our prior may approximate similar decision rules. These policies incorporate treatment history, outcome trends, and combinations of patient characteristics (definition details in Appendix D.8):
  • Deterioration: At step t, if the patient was not treated at t-1, start treatment if the target outcome increases relative to the previous step (deterioration); once treated, continue treatment.

  • Response-guided: At each step t, if the patient was not treated at t-1, start treatment when heart rate exceeds its 75th percentile; if the patient was treated at t-1, continue treatment while the target outcome is decreasing relative to the previous step, and stop it otherwise.

  • Tree: At each step t, treat if the target outcome increased relative to the previous step and the patient is older than the cohort median age, or if heart rate exceeds its 90th percentile; otherwise do not treat.

We compare PICLE-PFN with IGC-Net and D3, the two strongest baselines in our evaluation that support dynamic policies, across horizons of 2/4/6/8/10 with 20 repetitions. The table below summarizes RMSE averaged across horizons; results for individual horizons are provided in Table 6.

policyPICLE-PFNIGC-NetD3
deterioration0.515 ± 0.1340.567 ± 0.1330.542 ± 0.127
response-guided0.748 ± 0.1730.783 ± 0.1820.779 ± 0.162
tree0.908 ± 0.2340.979 ± 0.2740.975 ± 0.258

PICLE-PFN achieves the lowest mean normalized RMSE at every evaluated horizon under the 'deterioration' and 'response-guided' policies. Under the tree policy, no method wins at every horizon, but PICLE-PFN achieves the lowest mean normalized RMSE aggregated across horizons. These experiments broaden the evidence for generalization across dynamic policies.

W2 (continued)

  1. Generalization to unseen causal mechanisms. We expand our evaluation to six mechanism shifts outside or in the far tails of our pretraining distribution (Detailed descriptions of these shifts are in Appendix I).
  • Dense parents: increase the probability of dense parent sets to pdense=0.5p_{dense}=0.5 or 11, beyond the pretraining range [0,0.3][0,0.3].
  • Treatment tolerance/sensitization: make treatment effects depend directly on cumulative treatment exposure, a dependence mechanism absent from the prior.
  • Jump noise: introduce large, abrupt jumps to time-varying covariates, a noise distribution absent in the prior.
  • Extreme treatment assignment: use nearly deterministic, severity-driven treatment assignment with substantially reduced overlap.
  • Explosive dynamics: introduce geometrically growing untreated outcomes, an absent trend in our prior.
  • Explosive dynamics with limited overlap: combine the last two mechanisms, requiring counterfactual extrapolation into poorly supported regions.

Each setting has several configurations, ordered by increasing shift severity from the prior. For each configuration, we generate 20 tasks, each from its own randomly drawn DGP (graph, functions, noise, and policies). We then analyze PICLE-PFN's empirical CAPO coverage on all DGPs. A decline in empirical CAPO coverage would indicate that the predicted intervals become less reliable under distribution shift. The table below reports the average empirical CAPO coverage of the nominal 95% interval. Coverage remains broadly comparable to the corresponding in-prior configurations under most evaluated shifts. The clearest deterioration occurs when explosive dynamics are combined with limited overlap: coverage declines to 78.8% at h=10h=10, compared to 89.0% under limited overlap alone at the same horizon. Appendix I provides coverage curves across nominal levels and corresponding interval widths.

settingconfigurationτ\tau = 2τ\tau = 4τ\tau = 6τ\tau = 8τ\tau = 10
Densityp_dense = 0.3 (in-prior)0.9680.9760.9790.9800.968
p_dense = 0.50.9640.9680.9760.9760.960
p_dense = 1.00.9580.9570.9650.9680.965
Tolerance / Sensitizationno tolerance (in-prior)0.9660.9750.9790.9780.976
tolerance r = 50%0.9650.9730.9810.9820.981
tolerance r = 25%0.9660.9650.9810.9770.977
sensitization s = 200%0.9610.9760.9850.9870.988
sensitization s = 400%0.9660.9830.9900.9860.983
Jump noiseno jumps (in-prior)0.9660.9760.9810.9810.977
jump rate 0.10.9750.9820.9840.9880.989
jump rate 0.20.9780.9850.9770.9930.996
Extreme assignmentextreme assignment, always-treat0.9320.9650.9720.9690.972
Explosive effectstable, never-treat (anchor)0.9740.9790.9820.9840.985
explosive ρ = 1.30.9840.9830.9840.9830.988
Explosive + positivityextreme assignment only, never-treat (anchor)0.9540.9550.9430.9200.890
+ explosive ρ = 1.30.9340.9010.8830.8520.788

We further compare RMSE with IGC-Net and CausalTransformer in the combined setting, stratifying results by support level. PICLE-PFN retains competitive point-estimation accuracy under low support, but its reduced coverage reveals overconfidence.

supportPICLE-PFN 95 % coveragePICLE-PFN RMSECT RMSEIGC-Net RMSE
high0.95 ± 0.010.61 ± 0.090.55 ± 0.080.66 ± 0.12
low0.79 ± 0.041.12 ± 0.121.24 ± 0.131.27 ± 0.13

These experiments characterize robustness and limitations under the evaluated shifts. Generalization beyond the pretraining distribution and principled prior design remain active research questions for the PFN literature [1–3]. Extending these efforts to reliable longitudinal causal estimation, especially under limited support, is an important direction for future work.

[1] Müller, Samuel, et al. "Position: The future of bayesian prediction is prior-fitted." arXiv preprint arXiv:2505.23947 (2025).

[2] Zhang, Xiyuan, et al. "Mitra: Mixed synthetic priors for enhancing tabular foundation models." Advances in neural information processing systems 38 (2026): 15795-15840.

[3] Bouadi, Mohamed, et al. "Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Quality." arXiv preprint arXiv:2605.18971 (2026).

Thanks for your detailed response, I will keep my score as positive.

Reviewer_jS2mOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies longitudinal causal effect estimation by generating synthetic causal data and training a Transformer to predict policy-conditioned outcome distributions. The model supports dynamic treatment policies and reports strong performance without task-specific training.

Critical strengths

1.The method shows strong accuracy and fast deployment across several synthetic and semi-synthetic settings. 2. Supporting different policies and prediction horizons without retraining is practically useful.

Critical weaknesses

1.My main concern is novelty. CausalLongPFN already follows a very similar synthetic pretraining approach for longitudinal causal prediction. Dynamic policy conditioning and direct multi-step prediction are useful extensions, but their methodological novelty seems limited. 2.The gap between the synthetic pretraining distribution and real-world mechanisms remains a concern. The unseen-policy experiments provide some evidence, but do not establish robustness to broader mechanism shifts. The real-data experiment evaluates factual prediction rather than counterfactual estimation. 3.The comparison with CausalLongPFN is limited to Cancer because of checkpoint restrictions. This leaves the advantage over the closest method insufficiently characterized.

Questions for authors

1.What is the main methodological contribution beyond extending CausalLongPFN to dynamic policies and direct multi-step prediction? 2.Can the authors provide broader comparisons with CausalLongPFN under compatible input dimensions and prediction horizons? 3.How sensitive is the model to misspecification of the synthetic prior, particularly when deployment mechanisms differ substantially from those used during pretraining?

5 comments

W1 and Q1: We thank the reviewer for raising the question of methodological novelty. Although both methods use synthetic pretraining, they differ in their estimation procedure, policy representation, and architecture:

  1. Avoiding structural bias CausalLongPFN recursively applies a one-step predictor, inserting each predicted outcome mean into the next-step history while leaving future time-varying covariates masked. Under nonlinear dynamics, this procedure does not generally recover multi-step CAPO, even with a perfect one-step conditional mean model. PICLE-PFN instead learns directly from simulated multi-step counterfactual targets whose conditional expectation is CAPO, integrating over intermediate outcomes and covariates through simulation. Direct multi-step prediction therefore addresses a structural estimation limitation, rather than merely reducing the number of prediction steps. We provide an additional experiment using a simple DGP to illustrate this bias (see our response to W3 and Q2) and a mathematical explanation in Appendix E.

  2. Policy Representation CausalLongPFN conditions on a prescribed treatment sequence. A dynamic policy cannot generally be specified this way because its future actions depend on the evolving counterfactual history. PICLE-PFN instead represents the policy through its decisions on factual histories across the observational dataset. Proposition 3.1 establishes that, under the stated identification assumptions, these population-level demonstrations together with the observational distribution suffice to identify CAPO. This provides an identification foundation for a unified ICL interface across policy families, without requiring policy-specific parameterizations.

  3. Substantially faster inference through algorithmic and architectural design As shown in Appendix E, PICLE-PFN is about 40-300x faster than CausalLongPFN, whose runtime sometimes approaches that of task-specific baselines. Two design choices contribute to this difference. First, CausalLongPFN requires τ\tau forward passes to estimate a τ\tau-step-ahead CAPO due to its mean plug-in procedure, whereas PICLE-PFN requires only one. Second, CausalLongPFN allows context rows to attend to query rows, making context representations query-dependent. Consequently, the full context set must be re-encoded for each query trajectory and each rollout step. In contrast, PICLE-PFN prevents context tokens from attending to query tokens, allowing context representations to be computed once and reused across queries.

W3 and Q2: We first provide an additional MIMIC-based semi-synthetic DGP with compatible input dimensions, and then we show a synthetic DGP to demonstrate the mean plug-in bias introduced by CausalLongPFN's algorithm.

  • MIMIC-dim10. This benchmark contains 10 time-varying covariates and 2 binary treatments, compatible with CausalLongPFN's input dimension. (Detailed description in Appendix E)
HorizonRMSE (Ours)RMSE (CausalLongPFN)Runtime (s) (Ours)Runtime (s) (CausalLongPFN)
10.281 ± 0.0970.293 ± 0.0950.1977.69
20.380 ± 0.1110.425 ± 0.1230.21913.46
30.459 ± 0.1340.566 ± 0.1700.24419.72
40.533 ± 0.1390.705 ± 0.2310.26725.89
50.644 ± 0.2200.838 ± 0.2840.29032.15
  • Synthetic Experiment. CausalLongPFN's mean plug-in algorithm does not generally recover the multi-step CAPO. To illustrate, we create a simple synthetic DGP (Details in Appendix E) using function families (linear, softplus, sigmoid) in the prior of both CausalLongPFN and PICLE-PFN. We also create a reference for CausalLongPFN, where we replace its estimated mean for the autoregressive rollout step with the true mean value. This is what CausalLongPFN's procedure would return with a perfect one-step model, so its error comes from the plug-in procedure alone. The RMSE comparison below shows that, at τ=1\tau=1 both models perform similarly, since no predicted mean has been fed back yet. CausalLongPFN's error then grows with horizon, tracking the reference line, while PICLE-PFN's error increases only slightly. This is consistent with the theory: the mean plug-in algorithm introduces a plug-in bias.
HorizonPICLE-PFNCausalLongPFNPerfect mean plug-in
10.143 ± 0.0310.132 ± 0.0160.005
20.140 ± 0.0290.147 ± 0.0370.118
30.138 ± 0.0490.192 ± 0.0540.195
40.170 ± 0.0540.220 ± 0.0630.245
50.162 ± 0.0570.234 ± 0.0700.276

W2 and Q3 We expand our evaluation to six mechanism shifts outside or in the far tails of our pretraining distribution (Detailed descriptions of these shifts are in Appendix I).

  • Dense parents: increase the probability of dense parent sets to pdense=0.5p_{dense}=0.5 or 11, beyond the pretraining range [0,0.3][0,0.3].
  • Treatment tolerance/sensitization: make treatment effects depend directly on cumulative treatment exposure, a dependence mechanism absent from the prior.
  • Jump noise: introduce large, abrupt jumps to time-varying covariates, a noise distribution absent in the prior.
  • Extreme treatment assignment: use nearly deterministic, severity-driven treatment assignment with substantially reduced overlap.
  • Explosive dynamics: introduce geometrically growing untreated outcomes, an absent trend in our prior.
  • Explosive dynamics with limited overlap: combine the last two mechanisms, requiring counterfactual extrapolation into poorly supported regions.

Each setting has several configurations, ordered by increasing shift severity from the prior. For each configuration, we generate 20 tasks, each from its own randomly drawn DGP (graph, functions, noise, and policies). We then analyze PICLE-PFN's empirical CAPO coverage on all DGPs. A decline in empirical CAPO coverage would indicate that the predicted intervals become less reliable under distribution shift. The table below reports the average empirical CAPO coverage of the nominal 95% interval. Coverage remains broadly comparable to the corresponding in-prior configurations under most evaluated shifts. The clearest deterioration occurs when explosive dynamics are combined with limited overlap: coverage declines to 78.8% at h=10h=10, compared to 89.0% under limited overlap alone at the same horizon. Appendix I provides coverage curves across nominal levels and corresponding interval widths.

settingconfigurationτ\tau = 2τ\tau = 4τ\tau = 6τ\tau = 8τ\tau = 10
Densityp_dense = 0.3 (in-prior)0.9680.9760.9790.9800.968
p_dense = 0.50.9640.9680.9760.9760.960
p_dense = 1.00.9580.9570.9650.9680.965
Tolerance / Sensitizationno tolerance (in-prior)0.9660.9750.9790.9780.976
tolerance r = 50%0.9650.9730.9810.9820.981
tolerance r = 25%0.9660.9650.9810.9770.977
sensitization s = 200%0.9610.9760.9850.9870.988
sensitization s = 400%0.9660.9830.9900.9860.983
Jump noiseno jumps (in-prior)0.9660.9760.9810.9810.977
jump rate 0.10.9750.9820.9840.9880.989
jump rate 0.20.9780.9850.9770.9930.996
Extreme assignmentextreme assignment, always-treat0.9320.9650.9720.9690.972
Explosive effectstable, never-treat (anchor)0.9740.9790.9820.9840.985
explosive ρ = 1.30.9840.9830.9840.9830.988
Explosive + positivityextreme assignment only, never-treat (anchor)0.9540.9550.9430.9200.890
+ explosive ρ = 1.30.9340.9010.8830.8520.788

We further compare RMSE with IGC-Net and CausalTransformer in the combined setting, stratifying results by support level. PICLE-PFN retains competitive point-estimation accuracy under low support, but its reduced coverage reveals overconfidence.

supportPICLE-PFN 95 % coveragePICLE-PFN RMSECT RMSEIGC-Net RMSE
high0.95 ± 0.010.61 ± 0.090.55 ± 0.080.66 ± 0.12
low0.79 ± 0.041.12 ± 0.121.24 ± 0.131.27 ± 0.13

These experiments characterize robustness and limitations under the evaluated shifts. Generalization beyond the pretraining distribution and principled prior design remain active research questions for the PFN literature [1–3]. Extending these efforts to reliable longitudinal causal estimation, especially under limited support, is an important direction for future work.

[1] Müller, Samuel, et al. "Position: The future of bayesian prediction is prior-fitted." arXiv preprint arXiv:2505.23947 (2025).

[2] Zhang, Xiyuan, et al. "Mitra: Mixed synthetic priors for enhancing tabular foundation models." Advances in neural information processing systems 38 (2026): 15795-15840.

[3] Bouadi, Mohamed, et al. "Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Quality." arXiv preprint arXiv:2605.18971 (2026).

We would like to kindly follow up on our responses posted on November 14. We have further clarified our methodological novelty relative to CausalLongPFN, a concurrent work developed independently of ours. We have also conducted additional experiments to address your concerns regarding comparisons with CausalLongPFN under compatible settings and robustness to prior misspecification.

As the rebuttal period is approaching its end, we would greatly appreciate any further feedback on whether these additions address your concerns. Thank you again for your time and consideration.

Comments on the paper

1 comment

We thank all reviewers for their time and careful feedback. Before responding to individual comments, we want to disclose a correction to our experimental results.

After submission, we found implementation errors in the benchmark data preparation. These errors affected all methods, including PICLE-PFN. We fixed the pipeline, regenerated the benchmark data, and re-evaluated every method. Each baseline was re-tuned under the same protocol and budget as the original submission.

With the corrected pipeline, all methods' results changed. PICLE-PFN is still among the strongest methods in most settings, but its advantage is smaller than originally reported. In the updated results, PICLE-PFN achieves the lowest RMSE in six of seven benchmark settings, with up to 11.9% lower RMSE than the strongest baseline, while being about 100× to 5000× faster at deployment than a comparably accurate method. We are therefore revising our claim from "stronger performance" to "competitive performance", achieved without any per-task training or hyperparameter tuning. We have updated the results in the manuscript and released the corrected code. Appendix H reports the corrected results for the original checkpoint. The main text reports results after additional second-stage pretraining, introduced in response to the uncertainty-related comments from reviewers BLeE and 2ZnP. Point-estimation performance remains largely unchanged between the two checkpoints under the corrected pipeline (Appendix G); please see our response to reviewer BLeE’s W3 for details.

We apologize for the error and appreciate the opportunity to correct the record.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.