acceptodds
Under review as a conference paper at ICLR 2027

Multivariate Time Series Causal Foundational Model with In-Context Learning

Abstract

Predicting how a dynamic system evolves under a sequence of interventions is central to healthcare, economics, and engineering, and is complicated by time-varying confounding when treatments depend on history. Classical methods are trained from scratch per dataset and predict a single scalar outcome. Prior-data fitted networks (PFNs) enable training-free inference, but temporal PFNs either lack interventions or predict only one target variable, ignoring how the other covariates respond to treatment. We introduce MTS-CFM, a multivariate time series causal foundational model that forecasts every variable of the system, both the outcome and all covariates, over a multi-step horizon under a planned treatment sequence. Given an individual's history, a context set of related trajectories, and the treatment plan, it predicts the entire horizon in one forward pass without fine-tuning. MTS-CFM is trained entirely on synthetic trajectories sampled from a prior over temporal structural causal models with explicit treatment assignment. Without task-specific training, it performs competitively with domain-trained baselines and a causal PFN on interventional outcome prediction, while forecasting time-varying covariates more accurately than comparable baselines. These results make in-context learning a practical route to causal forecasting under dynamic treatments.

Primary area:
learning on time series and dynamical systems
Keywords:
prior-data fitted networks, in-context learning, time series, causal inference, foundational models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hH98OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper applies the prior-fitted-networks, a popular amortized foundational model, to the problem of predicting longitudinal potential outcome under a sequence of interventions. The network takes the historical dataset as input, and predict the CID of all the variables in the future. Results on synthetic and semi-synthetic datasets are promising.

Critical strengths
  1. This paper tackles an important problem. The architecture is sound in that it uses mask mechanism to ensure the correct information flow.

  2. Unlike previous PFNs, which usually predict a single step / a scalar target, it ambitiously predicts all the future covariates in one pass under fixed interventions and shows its effectiveness.

Critical weaknesses
  1. The training data and the causal quantity of interest are misaligned. The pre-training dataset includes 30% observational trajectories. These trajectories inject confounding to the network.

  2. Prior domain knowledge tells us, a generic network does not work well for longitudinal counterfactual outcome prediction. This is the reason why G-Net, RMSN, G-transformer explicitly incorporate the confounding adjusting into the architecture. CLPFN does g-computation via recursive one-step prediction. This paper, however, does not have the design that can address the time-varying confounding issue (the mask can do partly, but not enough for causal adjustment). Yet, it claims superiority over existing baselines without proper justifications.

  3. The experiment is weak. The experiment does not answer the question: is the gain from adjusting for time-varying confounding? In addition, Ablation studies over key designs are also missing.

Questions for authors

See weaknesses above.

Reviewer_WAm6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MTS-CFM, a pre-trained model for forecasting the outcome and time-varying covariates under a planned treatment sequence. It trains a transformer on synthetic trajectories from a prior over temporal structural causal models and then makes predictions with in-context examples, without fitting new weights for each dataset. Attention is applied along the time, variable, and cohort axes. The experiments cover three mechanistic simulators with intervention ground truth and MIMIC-III for factual prediction. MTS-CFM performs well against longitudinal causal estimators and a causal PFN, particularly for multi-step covariate forecasts.

Critical strengths
  1. Predicting all covariates as well as the outcome is a worthwhile extension of longitudinal causal PFNs. In settings such as HIV and warfarin, treatment changes other parts of the system, so evaluating only the outcome misses an important part of the forecasting task.

  2. The evaluation includes both zero-shot and dataset-trained competitors, and the authors explicitly separate simulated interventional evaluation from factual prediction on MIMIC-III. The per-covariate results in the appendix are useful for understanding where the gains occur.

Critical weaknesses
  1. There is a mismatch between the stated distributional target and the predictive head. Section 4.1 factorizes the forecast density into univariate distributions for each variable and horizon step. Shared hidden representations can improve these marginals, but the product distribution cannot express conditional dependence between future variables or across time. The claims about forecasting the interventional distribution of the entire system therefore appear stronger than what is actually evaluated.

  2. The experiments show low error under planned treatments, but do not isolate whether the model responds correctly to changes in treatment plans. The reported RMSE aggregates many trajectories and plans; a model that relies heavily on patient history could look good even if it underestimates the difference between two interventions. The real-data results cannot settle this point because MIMIC-III is evaluated only on factual trajectories.

Questions for authors
  1. On the three simulated domains, could the authors evaluate treatment contrasts for the same patient history, such as the predicted change in outcome and covariates between two specified future plans? A comparison with the simulator's corresponding contrast would test intervention sensitivity more directly than pooled RMSE. A simple shuffled-plan or masked-plan control would also be informative.

  2. Section 4.1 writes the predictive distribution as a product of per-variable and per-step mixtures. Do the authors intend to claim only calibrated interventional marginals rather than a joint trajectory distribution? If the latter is intended, how would correlations across future covariates and time steps be represented or sampled?

  3. Could the authors include an ablation of variate attention, with similar parameter count and training budget, on the HIV and warfarin multi-step tasks? Table 2 shows that multivariate prediction is effective, but does not show how much of that gain is specifically due to joint variate modeling rather than the larger pre-training setup.

Reviewer_c5PcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MTS-CFM, a prior-data fitted network (PFN) trained entirely on synthetic temporal structural causal models. It predicts both the outcome and time-varying covariates over several future steps under a given treatment plan. The inputs are an individual's history, static covariates, related context trajectories, and the treatment plan, and the model makes its predictions in one forward pass without task-specific fine-tuning. It uses attention across time, variables, and context trajectories, with a univariate Gaussian-mixture head for each target cell. The temporal SCM prior comes from CLPFN. The experiments use three simulators for interventional evaluation and MIMIC-III for factual prediction. The clearest reported gains are on HIV and multi-step covariate forecasting.

Critical strengths
  1. The full-system forecasting setup is useful. Instead of looking only at a single outcome, MTS-CFM also predicts how the covariates respond to treatment. This gives a broader picture of what happens under a treatment plan. Doing this over several steps in one pass, without fine-tuning on the target dataset, makes the setup practically appealing. The three-axis attention and masking design provide a concrete way to support this interface.
  2. Some of the reported improvements are substantial. For example, Table 1 gives HIV horizon-5 outcome RMSE of 0.159, compared with 0.288 for CLPFN. In Table 2, horizon-5 covariate RMSE is about 39%, 31%, and 6% lower than CLPFN on HIV, warfarin, and MIMIC-III, respectively. These results make a useful case for predicting multiple variables together. It is also good that the authors clearly separate interventional tests on simulators from factual prediction on MIMIC-III, rather than treating the latter as direct evidence of causal validity.
Critical weaknesses
  1. The factual/interventional training mix needs a clearer explanation. Section 4.2 uses 70% intervention-resimulated query examples and 30% factual continuations. What is not clear is how the factual examples are represented relative to an intervention plan. If their realised future actions are passed in the same way as a fixed plan, this could matter for multi-step prediction: under an adaptive policy, later actions depend on intermediate future states, so conditioning on those actions can select those states. The resulting conditional distribution need not match the intervention distribution in Eq. (1). I would like a clearer explanation of how the inputs, masks, or targets distinguish the two cases, and why the mixed objective still learns the intended causal quantity. I am raising this as an important point to clarify, not as a claim that the implementation is wrong.
  2. It is still hard to tell where the multivariate gains come from. The model reuses CLPFN's temporal SCM prior, so a comparison using the same prior and training budget would help separate the benefit of predicting all targets together from the benefit of the attention design or other training choices. The current comparisons with G-Net and a relabelled single-target CLPFN are useful, but do not make that distinction. Also, Table 3 only evaluates one-step outcome marginals, so it does not tell us enough about the probabilistic quality of the main multi-step, all-variable task. The RMSE results are promising, but these gaps make the main contribution harder to assess.
Questions for authors
  1. How are the factual training examples handled? For the 30% factual query examples, can the model see the realised future treatments? How can it tell these examples apart from planned interventions? Could you explain why the mixed objective still targets Eq. (1)? If the two cases use the same input representation, a limited comparison with 100% intervention-supervised query examples would help test this concern.
  2. Am I right that the outputs are marginals, not a joint trajectory distribution? Shared tokens can help with marginal prediction, but a product of univariate mixtures does not itself capture dependence between the predicted variables once the inputs are fixed. The limitations section already notes that the model cannot sample coherent joint trajectories. Could you make the formal target and the statements about dependence consistent with this? I am not asking for a new joint model here; a clear statement of what the current model predicts would address the point.
  3. Could you provide a focused control for the main design choice? A small comparison with the same prior and training budget could show how much comes from all-target supervision and how much comes from the attention design. Results on multi-step covariate probabilistic quality would also help support the predictive-distribution claims. A focused check of these existing claims would be enough; I am not asking for a broad new benchmark suite.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.