acceptodds
Under review as a conference paper at ICLR 2027

HiSTOR: Structured Treatment-outcome Representation Learning for High-Dimensional Longitudinal Data

Abstract

Estimating potential outcomes under alternative treatment sequences is essential for personalized decision-making, but remains challenging with high-dimensional covariates and time-varying confounding. In this study, we propose HiSTOR (High-dimensional Structured Treatment-Outcome Representation), an end-to-end framework that integrates nonlinear dimension reduction with longitudinal counterfactual prediction. The central idea is to learn structured low-dimensional representations of both current patient states and evolving histories, according to their shared and distinct relationships with treatment assignment and outcomes. This extends latent-space covariate adjustment to longitudinal settings, jointly addressing covariate dimensionality and historical complexity. The resulting framework supports one-step and multi-step outcome prediction under specified treatment sequences. Experiments on CVSim and CancerSim demonstrate the superiority of HiSTOR across varying covariate dimensions, confounding strengths, and prediction horizons compared to state-of-the-art methods. To sum up, HiSTOR offers a unified approach to connecting high-dimensional covariate adjustment with longitudinal potential-outcome prediction. The code and tutorials are available at https://github.com/anonymous.

Primary area:
causal reasoning
Keywords:
Counterfactual Prediction, longitudinal causal inference, counterfactual outcome, causal machine learning, High-Dimensional Longitudinal Data

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oWSfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HiSTOR for longitudinal potential-outcome prediction with high-dimensional covariates. It combines a current-covariate encoder, a causal temporal Transformer, and a four-part latent representation for shared, outcome-specific, treatment-specific, and residual-reconstruction information. Factual outcome prediction, auxiliary treatment prediction, reconstruction, adversarial distribution matching, and an optional rollout loss are used for training, with one-step, autoregressive, and direct multi-step prediction. Experiments cover CancerSim, CVSim, and a semi-synthetic MIMIC-III study. However, the main components are established causal representation-learning and longitudinal EHR techniques. The CVSim extension adds noisy measurements of an already observed low-dimensional factor rather than new confounding information.

Critical strengths
  1. The paper introduces a structured representation framework for high-dimensional longitudinal causal inference, encoding the current patient state and prior history by their treatment and outcome roles, and supports both one-step and multi-step treatment-conditioned prediction.

  2. The empirical evaluation covers varying confounding strengths, covariate dimensions, and prediction horizons, with HiSTOR reporting the lowest mean errors across the tested CancerSim and CVSim settings.

Critical weaknesses
  1. The proposed method largely recombine established components: causal temporal Transformers, time embeddings, shared/treatment/outcome latent decomposition, task-specific heads, adversarial separation, and reconstruction. These ideas already appear in existing studies. The paper does not show that the four-branch routing outperforms a shared multi-task representation or introduces a new identification mechanism. The novelty is somewhat weak.

  2. The four latent branches are slices of one MLP output and are separated only by which heads read them. There is no independence, orthogonality, balancing, or sufficiency constraint, so z_t^0 may omit confounders or duplicate information stored elsewhere. The method still depends on unmeasured-confounding assumptions.

  3. The “w/o Reconstruction” ablation does not isolate the residual branch z_t^3; it removes the reconstruction objective itself. Since the direct decoder uses all four branches, the performance change cannot be attributed specifically to z_t^3. This ablation should preserve both the reconstruction loss and parameter count while removing only the residual branch.

  4. The loss weighting hyperparameters appearing in the equations are fixed without sensitivity analysis. In particular, λX\lambda_X, λrec\lambda_{\mathrm{rec}}, λadv(e)\lambda_{\mathrm{adv}}(e), λnorm\lambda_{\mathrm{norm}}, λroll\lambda_{\mathrm{roll}}, and the gradient-penalty weight λgp\lambda_{\mathrm{gp}} each use one setting, while only the adversarial schedule is specified.

Questions for authors

Can the authors provide a real-data off-policy evaluation using target-trial emulation or a doubly robust estimator, including overlap diagnostics and sensitivity analysis? If this is not feasible, please state explicitly that the real MIMIC-IV evaluation is factual forecasting only and does not establish treatment-effect validity.

4 comments
AuthorsOpenReview ↗

Reviewer3_part1of4

Response to Reviewer 3 (1/4)

We thank the reviewer for the careful reading. We clarify each point below. All new experiments reported below use the submitted model, and the main result is unchanged by them: HiSTOR has the lowest five-horizon error in all 23 simulation settings of the paper (18 CVSim, 5 CancerSim). Unless stated otherwise, every new number is NRMSE (%) averaged over five seeds, every relative change is paired by seed against the submitted model trained in the same batch of runs, and * marks a change whose 95% paired t-interval excludes zero. "One-step" is the one-step-ahead error and "five-horizon" is the mean over the five multi-step horizons, as in the paper. In these reruns the submitted model obtains 0.666 / 0.904 on CancerSim γ\gamma = 4, 0.876 / 1.150 on CVSim Static dhdd_{\mathrm{hd}} = 2000 and 0.845 / 1.158 on CVSim Temporal dhdd_{\mathrm{hd}} = 2000 (one-step / five-horizon), against 0.670 / 0.911, 0.882 / 1.197 and 0.845 / 1.167 in the paper's tables.

W1. The method recombines existing components. No evidence that four-branch routing beats a shared representation.

None of CRN, CT, Causal CPC or IGC-Net compresses the current high-dimensional state separately from the history, and none is competitive on both tasks at dhdd_{\mathrm{hd}} ≥\geq 200. HiSTOR's combination (separate compression of the current state, reconstruction-retained compression with a residual slot, and a frozen-representation, plan-conditioned direct decoder) is what yields the lowest five-horizon error in all 23 simulation settings, with margins of 22-28% over the best baseline for dhdd_{\mathrm{hd}} ≥\geq 200.

The question of routing versus a shared representation is answered by the equal-width comparison in which every head reads the whole latent, run in eight data settings with three paired seeds. Under the paper's loss weights the two are statistically indistinguishable, so routing costs no accuracy. With a strong treatment term (λX\lambda_{X} = 1 instead of the submitted 0.01) on CVSim Static at dhdd_{\mathrm{hd}} = 2000 with the patient factor withheld from all models, routing is 10.0% better over five horizons and 4.7% better one-step (paired intervals exclude zero). The components that the new controls isolate account for the accuracy. At dhdd_{\mathrm{hd}} = 2000, removing the history encoder raises the five-horizon error by 44.7% (Static) and 15.1% (Temporal), removing the current-state path raises the one-step error by 27.5% and 30.8% and the five-horizon error by 15.8% and 12.4%, and autoregressive decoding is 4.8% and 5.5% worse than direct decoding over five horizons.

AuthorsOpenReview ↗

Reviewer3_part2of4

Response to Reviewer 3 (2/4)

W2. The branches are slices of one MLP output without independence, orthogonality or sufficiency constraints.

An independence or orthogonality penalty between the slots is deliberately not part of the method, for three reasons.

First, it is not what the estimand requires. Under sequential ignorability (Eq. 3), a representation supports valid adjustment if it is sufficient for the history with respect to the outcome (a prognostic score, Hansen, 2008) or with respect to treatment assignment (a balancing score, Rosenbaum and Rubin, 1983). Both are sufficiency conditions on what a slot contains, and neither involves independence between slots. In HiSTOR these conditions are targeted directly by the supervision: the outcome loss trains (z0z_0, z1z_1) to be predictive of the outcome and the treatment loss trains (z0z_0, z2z_2) to be predictive of treatment.

Second, statistical independence between latent blocks is not identifiable from observational data without further inductive bias or supervision (Locatello et al., 2019), and linear orthogonality can always be obtained by a rotation of the latent without changing any prediction. A penalty of this kind would therefore enforce a property that cannot be verified and that has no bearing on the estimand.

Third, such penalties can remove information that adjustment needs. Representations constrained to be invariant or balanced are in general not invertible, and the information they discard induces confounding bias (Johansson et al., 2019, Melnychuk et al., 2024). HiSTOR takes the opposite route and keeps the latent information-preserving: the reconstruction objective and the residual slot z3z_3 retain covariate information that neither head needs, and the plan decoder reads all four slots, so nothing is discarded from the latent.

The design is thus structure learning by routing: the outcome loss can only shape (z0z_0, z1z_1), the treatment loss only (z0z_0, z2z_2), and the reconstruction residual only z3z_3, so the supervision signals are assigned to different subspaces by construction, which makes each head's problem easier to learn than with one shared vector. The adversarial term complements this at the level of the covariate code: it matches the compressed code to an isotropic Gaussian prior, which pushes its coordinates towards independence. On confounders missed by z0z_0: z0z_0 is read by both heads and is the only slot trained by both losses, so information relevant to both treatment and outcome is what it is trained to carry. Like all methods compared, HiSTOR relies on sequential ignorability (Eq. 3). This is an assumption of the setting, not of our method. We will add this discussion to the appendix.

W3. "w/o Reconstruction" removes the whole reconstruction objective and cannot isolate z3z_3. The decoder reads all four branches.

We ran both controls the reviewer suggests: (i) the same frozen backbone with a decoder that reads only

$z_0$, $z_1$, $z_2$

, and (ii) a backbone without the residual slot, with the 64 dimensions reallocated so that the total width is unchanged. For (i), the five-horizon error rises by 0.7% on CancerSim (γ\gamma = 4) and by 4.5% and 1.5% on CVSim Static and Temporal (dhdd_{\mathrm{hd}} = 2000), none significant (five paired seeds). For (ii), it changes by -4.0% on CancerSim (paired interval excludes zero) and by +3.9% and -5.3% on CVSim Static and Temporal (not significant). The residual slot therefore matters little where the covariates are low-dimensional, as on CancerSim, and we will report both controls next to the existing "w/o Reconstruction" row so that the three effects can be read separately.

Model (one-step / five-horizon)CancerSim γ\gamma = 4CVSim Static dhdd_{\mathrm{hd}} = 2000CVSim Temporal dhdd_{\mathrm{hd}} = 2000
HiSTOR0.666 / 0.9040.876 / 1.1500.845 / 1.158
ii Decoder reads
$z_0$, $z_1$, $z_2$

only | 0.666 / 0.911 (+0.7%) | 0.876 / 1.201 (+4.5%) | 0.845 / 1.176 (+1.5%) | | iiii Backbone without the residual slot | 0.653 (-1.9%) / 0.868 (-4.0%*) | 0.889 (+1.5%) / 1.195 (+3.9%) | 0.857 (+1.4%) / 1.097 (-5.3%) |

In (i) the backbone is frozen and shared with HiSTOR, so the one-step error is identical. In (i) on CVSim Static the increase grows with the horizon (-4.2%, -0.1%, +6.6%, +6.5% and +10.7% at τ\tau = 1, …, 5).

AuthorsOpenReview ↗

Reviewer3_part3of4

Response to Reviewer 3 (3/4)

W4. No sensitivity analysis of the loss weights.

We agree that the paper did not list all loss weights and reported no sensitivity analysis. we list all weights below, including λrec\lambda_{\mathrm{rec}} = 1,together with a sensitivity study. λgp\lambda_{\mathrm{gp}} = 10 with three critic updates is the WGAN-GP default (Gulrajani et al., 2017), and λadv\lambda_{\mathrm{adv}} = λnorm\lambda_{\mathrm{norm}} = 0.01 are used unchanged on both simulators. We ran a one-at-a-time study over 0.1$\times$ and 10$\times$ of each weight and three alternative branch-width allocations, on CancerSim γ\gamma = 4 and CVSim dhdd_{\mathrm{hd}} = 2000 Static with five paired seeds. Branch widths (16/16/32/96, 32/32/64/32, 40/40/40/40) change the five-horizon error by -4.1% to +3.3%. Across all loss weights the five-horizon error changes by -6.0% to +13.0% (CancerSim) and -2.1% to +12.8% (CVSim). The largest increases (11-13%) come from the tenfold reconstruction and adversarial weights, λgp\lambda_{\mathrm{gp}} = 1 and λX\lambda_{X} = 1 on CVSim, and the only increase outside its paired interval is the tenfold adversarial weight on CancerSim (+4.2%). Every configuration keeps a lower five-horizon error than the best baseline in the paper's tables (at most 1.02 vs. 1.07 on CancerSim and 1.30 vs. 1.67 on CVSim). At one step the same holds except for the two tenfold weights on CVSim (1.01 and 1.05 vs. 0.95 for IGC-Net). The full table will be added to the appendix.

Changed settingCancerSim γ\gamma = 4, one-stepCancerSim γ\gamma = 4, five-horizonCVSim Static dhdd_{\mathrm{hd}} = 2000, one-stepCVSim Static dhdd_{\mathrm{hd}} = 2000, five-horizon
Submitted0.6660.9040.8761.150
λadv\lambda_{\mathrm{adv}} = 0.0010.660 (-0.9%)0.878 (-2.8%)0.891 (+1.7%)1.232 (+7.1%)
λadv\lambda_{\mathrm{adv}} = 0.10.733 (+10.0%*)0.942 (+4.2%*)1.049 (+19.8%)1.280 (+11.3%)
λnorm\lambda_{\mathrm{norm}} = 0.0010.687 (+3.1%)0.920 (+1.7%)0.833 (-0.5%)1.220 (+8.9%)
λnorm\lambda_{\mathrm{norm}} = 0.10.668 (+0.2%)0.923 (+2.1%)0.902 (+2.9%)1.239 (+7.8%)
λrec\lambda_{\mathrm{rec}} = 0.10.685 (+2.8%)0.875 (-3.3%)0.880 (+0.5%)1.125 (-2.1%)
λrec\lambda_{\mathrm{rec}} = 100.710 (+6.6%)1.022 (+13.0%)1.010 (+15.3%)1.297 (+12.8%)
λgp\lambda_{\mathrm{gp}} = 10.668 (+0.3%)0.906 (+0.2%)0.846 (+1.0%)1.253 (+11.8%)
λX\lambda_{X} = 00.647 (-2.9%)0.893 (-1.2%)0.897 (+2.4%)1.164 (+1.3%)
λX\lambda_{X} = 0.001n/an/a0.883 (+0.8%)1.238 (+7.6%)
λX\lambda_{X} = 0.010.642 (-3.6%*)0.850 (-6.0%*)submittedsubmitted
λX\lambda_{X} = 0.10.662 (-0.6%)0.869 (-3.9%)0.884 (+0.9%)1.141 (-0.7%)
λX\lambda_{X} = 1submittedsubmitted0.943 (+7.7%)1.279 (+11.3%)
λroll\lambda_{\mathrm{roll}} = 0submittedsubmitted0.853 (-2.6%*)1.171 (+1.8%)
λroll\lambda_{\mathrm{roll}} = 0.1n/an/a0.873 (-0.4%)1.235 (+7.4%)
λroll\lambda_{\mathrm{roll}} = 0.5n/an/a0.920 (+5.1%*)1.238 (+7.7%)
Widths 16/16/32/960.649 (-2.5%)0.873 (-3.5%)0.833 (-0.5%)1.133 (+1.1%)
Widths 32/32/64/320.652 (-2.1%)0.867 (-4.1%)0.844 (+0.8%)1.157 (+3.3%)
Widths 40/40/40/400.671 (+0.7%)0.880 (-2.6%)0.845 (+0.8%)1.132 (+1.0%)

Submitted values: λadv\lambda_{\mathrm{adv}} = λnorm\lambda_{\mathrm{norm}} = 0.01, λrec\lambda_{\mathrm{rec}} = 1, λgp\lambda_{\mathrm{gp}} = 10, widths 24/24/48/64. λX\lambda_{X} = 1, λroll\lambda_{\mathrm{roll}} = 0 on CancerSim and λX\lambda_{X} = 0.01, λroll\lambda_{\mathrm{roll}} = 0.25 on CVSim. Relative changes are paired by seed. On CVSim the rows for λnorm\lambda_{\mathrm{norm}} = 0.001, λgp\lambda_{\mathrm{gp}} = 1 and the three width allocations use four seeds, so their means are not directly comparable with the five-seed submitted row, while the relative changes are.

AuthorsOpenReview ↗

Reviewer3_part4of4

Response to Reviewer 3 (4/4)

Q1. Off-policy evaluation on real data (MIMIC-IV).

Our clinical study uses MIMIC-III covariates with semi-synthetic treatments and outcomes (the IGC-Net generator), not MIMIC-IV. Appendix I states that it "is a semi-synthetic counterfactual benchmark, not a real-outcome clinical prediction experiment". This is the standard protocol of CT, Causal CPC and IGC-Net, and it exists because counterfactual accuracy cannot be scored on real outcomes: off-policy estimates on real data are themselves model-based and would add unverifiable assumptions to the comparison. The semi-synthetic design keeps real covariate complexity while retaining ground truth. We will make the semi-synthetic nature of this study more prominent in the main text, and list target-trial emulation with overlap diagnostics as future work.

We will update the revised paper with all of the clarifications and new results above.

Reviewer_na28OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

HiSTOR develops a machinery to predict counterfactual potential outcome in the framework of longitudinal treatment sequences using high-dimensional measurements of patient’s characteristics. It combines two existing encoders, CausalEGM and causal transformer, to extract feature representations from patient’s current and historical data, respectively as part of prediction model building. Both one-time-unit ahead prediction and multi-time-units ahead predictions are studied, where the latter consists of an autoregressive prediction and a decoder-based prediction under a prefixed treatment plan. The authors demonstrate strong empirical performance of HiSTOR across scenarios with varying confounding strengths, covariate dimensions, and look-ahead scenarios on benchmarks CancerSim and modified CVSim.

Critical strengths

Integrating current measurements and accrued history enables the use of rich temporal information from longitudinal data. Using separate CausalEGM and causal transformer to extract feature representations from patient’s current and historical data, respectively, can leverage neural-network architectural flexibility in the building of prediction models.

Critical weaknesses
  1. Fundamentally, counterfactual or potential outcome is the primary terminology preserved in causal inference. If causality is not the focus of this paper, which seems to be the case, then potential outcome should not be used in the presentation of the work, which once again is exclusively preserved for the causal inference. Indeed, this term “potential outcome” causes much confusion in the understanding of this paper. Moreover, causality requires a stringent control of confounding factors, which cannot be easily realized in observational studies, even through the use of propensity score approach (e.g. IPW) is used in part of estimation. Equation (2) defines both one-time-unit and multi-time-unit estimands of intervention effects, while Equations (7) and (9) specify their respective neural-network-based predictors. These predictors raise two concerns. One concerns the identifiability issue of expressing causal estimands via the observed-data distribution, which is of critical importance for the uncertainty quantification. The other one pertains to whether the proposed estimation actually targets the intervention estimand. In fact, the loss function based justification does not provide convincing arguments for the recovery of the true estimand.
  2. It is not clear that improvement of prediction accuracy on each neural-network component is attributed to the proposed constructs or the existing neural-network structures. The anonymous repository link to GitHub is invalid so it is impossible to use part of coding scripts available in the Supplementary Material to check thoroughly and wholly the computational aspects, including simulation and data results. The simulation experiments did not show clearly attributed benefits to the two encoders over existing neural-network components. In addition, some simulation settings are not well formulated to ensure fair comparison. For example, Appendix F specifies a four-layer, width-128 Transformer for HiSTOR, but a competing model of a one-layer CT used in the comparison has much smaller CVSim widths. In addition, Table 9 lists different checkpoint-selection criteria across different comparisons, which may restrain the validity of simulation results. The ablation study is not well presented with no clear evidence relevant related to the direct advantage of combining two encoders in the evaluation of treatment estimands. So, this empirical study is not convincing to demonstrate the proposed method.
Questions for authors
  1. How do you address the identifiability issue of expressing causal estimands via the observed-data distribution and uncertainty quantification?

  2. How do you ensure the proposed estimation actually targets the intervention estimand?

  3. The novel contribution in this paper seems to lie in the combination of two encoders for feature representations related respectively to current state and history. How is the simulation or ablation study used to reflect the attributed benefits of such encoders over the existing neural-network structures?

  4. There are lacking details in the specification of the architecture and loss changes for each variant in Table 4, including the replacement for the covariate bottleneck and the handling of connected reconstruction, adversarial, and residual-branch modules. Which loss terms and parameter updates are disabled? Are the remaining architecture and training settings held fixed? Have you designed a comparison study using the same backbone, covariate encoder, direct decoder, and checkpoint selection but an equally sized unpartitioned representation? In addition, baseline tuning and selection procedures are not well presented.

  5. In the code provided, it seems that the non-finite results have been removed. It is not unclear if any non-finite predictions ever occurred. It is important to report excluded and evaluated counts related to prediction failures. If prediction failures occurred, the paper needs to report them and provide a failure-aware comparison.

6 comments
AuthorsOpenReview ↗

Reviewer2_part1of4

Response to Reviewer 2 (1/4)

We thank the reviewer for the careful reading. We clarify each point below. All new experiments reported below use the submitted model, and the main result is unchanged by them: HiSTOR has the lowest five-horizon error in all 23 simulation settings of the paper (18 CVSim, 5 CancerSim). Unless stated otherwise, every new number is NRMSE (%) averaged over five seeds, every relative change is paired by seed against the submitted model trained in the same batch of runs, and * marks a change whose 95% paired t-interval excludes zero. "One-step" is the one-step-ahead error and "five-horizon" is the mean over the five multi-step horizons, as in the paper. In these reruns the submitted model obtains 0.666 / 0.904 on CancerSim γ\gamma = 4, 0.876 / 1.150 on CVSim Static dhdd_{\mathrm{hd}} = 2000 and 0.845 / 1.158 on CVSim Temporal dhdd_{\mathrm{hd}} = 2000 (one-step / five-horizon), against 0.670 / 0.911, 0.882 / 1.197 and 0.845 / 1.167 in the paper's tables.

W1. "Potential outcome" terminology, control of confounding, identifiability and the target of the estimator.

The paper is causal throughout. The estimands in Eq. (2) are interventional quantities in the longitudinal potential-outcomes framework, and Eq. (3) states the identifying assumptions (consistency, positivity, sequential ignorability). This is the setting and terminology of CRN, CT, Causal CPC and IGC-Net. Our wording (potential outcomes, counterfactual outcome prediction under a treatment sequence, time-varying confounding) and our problem setup are taken from this line of work without change, so that the formulation and the results are directly comparable with it. Because these assumptions cannot be verified on observational data, every quantitative claim is evaluated against simulator-generated counterfactual outcomes, which is the only way to measure interventional accuracy. We accept that the causal scope is stated too late in the submitted text, and we will move it to the first paragraph of Section 3. The two specific concerns raised here, identifiability together with uncertainty quantification and whether the estimator targets the interventional estimand, are answered under Q1 and Q2.

W2. Attribution of the gains, code availability and fairness of the comparison.

Are the gains due to the proposed constructs or to existing networks?

Both paths that define HiSTOR are necessary. At dhdd_{\mathrm{hd}} = 2000, removing the history encoder raises the five-horizon error by 44.7% (Static) and 15.1% (Temporal), and removing the current-state path raises the one-step error by 27.5% and 30.8%. The full controls, with all three models at the same parameter count, are given under Q3.

The anonymous code link does not work.

We apologise for the confusion: we marked the link as a placeholder. The URL in the abstract is a placeholder, as double-blind review requires: a public repository would identify the authors. The code itself is provided in the supplementary material, which contains the complete HiSTOR implementation, both simulators, the configuration files and the scripts that run the experiments. The method and its results can therefore be inspected and reproduced from the submission without any identifying information. The full repository will be made public once the review process has concluded. Until then, the supplementary material is the reference for the implementation.

AuthorsOpenReview ↗

Reviewer2_part2of4

Response to Reviewer 2 (2/4)

(W2 continued)

Capacity: HiSTOR uses a 4-layer, width-128 Transformer, CT one layer.

CT runs with its authors' tuned per-γ\gamma configurations on CancerSim and, on CVSim, where no official configuration exists, with its official γ\gamma = 0 architecture. CT's own search space fixes one layer and two heads. Capacity does not explain the gap: RMSN has 1.8M parameters at dhdd_{\mathrm{hd}} = 2000 and its five-horizon error is 3.40 against 1.20 for HiSTOR, and 82% of HiSTOR's parameters at dhdd_{\mathrm{hd}} = 2000 are in layers whose size is set by the input width (1.94M parameters at dhdd_{\mathrm{hd}} = 0, 10.72M at dhdd_{\mathrm{hd}} = 2000), not in the sequence model. To settle the question directly we ran (i) CT with HiSTOR's Transformer size (4 layers, width 128, 4 heads), (ii) CT with its official high-dimensional configuration, and (iii) HiSTOR reduced to the baselines' parameter budget (width 32, one Transformer layer). Five paired seeds:

ModelParameters (CVSim dhdd_{\mathrm{hd}} = 2000 / CancerSim)CVSim StaticCVSim TemporalCancerSim γ\gamma = 4
CT (submitted configuration)48K / 7.2K1.178 / 2.2431.126 / 1.9041.018 / 1.154
CT, official high-dimensional configuration84K / 21K1.109 / 2.1111.052 / 1.9580.890 / 1.065
CT-large (4 layers, width 128, 4 heads)3.68M / 1.90M1.072 / 1.6821.083 / 1.7065.739 / 2.049
HiSTOR-small186K / 57K1.113 / 1.4421.085 / 1.3860.723 / 0.924
HiSTOR10.72M / 1.93M0.876 / 1.1500.845 / 1.1580.666 / 0.904

Entries are one-step / five-horizon NRMSE.

Giving CT HiSTOR's Transformer size lowers its CVSim error by 25% and 10% but leaves it 46-47% above HiSTOR, and on CancerSim the enlarged CT is far worse than the tuned one. HiSTOR-small, with 5% of CT-large's parameters, has a lower five-horizon error than every CT variant in all three settings (57K parameters on CancerSim). At one step the full model is 18-20% below the best CT variant on CVSim. Capacity therefore does not account for the gap to CT. The parameter counts of all compared configurations are given in the table above.

Checkpoint selection differs across methods (Table 9).

Table 9 reports each method's official protocol. CT, CRN and RMSN train for their fixed budget and use the final (EMA) weights, as in their reference implementations. For IGC-Net and Causal CPC we added best-validation restoration, which can only help them. No method uses test or counterfactual data for selection. We also retrained CT, CRN and RMSN with best-validation restoration (patience 30, CancerSim γ\gamma = 4 and CVSim dhdd_{\mathrm{hd}} = 2000 Static/Temporal, five paired seeds). Their five-horizon error changes by -6% to +3% for CT, -21% to -3% for CRN and -14% to -3% for RMSN, mostly within the paired intervals. The ranking is unchanged and HiSTOR's margin over the best of the three after restoration remains 18% (one-step) and 15% (five-horizon) on CancerSim, and 25% and 41-50% on CVSim (five-horizon 1.15 and 1.16 for HiSTOR against 2.28 and 1.97 for CT with restoration).

Model (one-step / five-horizon)CancerSim γ\gamma = 4CVSim Static dhdd_{\mathrm{hd}} = 2000CVSim Temporal dhdd_{\mathrm{hd}} = 2000
CT, final weights1.018 / 1.1541.178 / 2.2431.126 / 1.904
CT, best-validation restoration0.912 (-10.4%*) / 1.080 (-6.4%)1.166 (-1.0%) / 2.280 (+1.6%)1.127 (+0.0%) / 1.967 (+3.3%)
CRN, final weights0.844 / 1.0964.345 / 7.8814.565 / 8.994
CRN, best-validation restoration0.861 (+1.9%) / 1.059 (-3.4%)4.163 (-4.2%) / 6.717 (-14.8%)3.713 (-18.7%) / 7.128 (-20.8%)
RMSN, final weights0.824 / 1.1102.907 / 3.6282.655 / 2.718
RMSN, best-validation restoration0.815 (-1.1%) / 1.078 (-2.9%*)2.722 (-6.4%*) / 3.228 (-11.0%)2.549 (-4.0%) / 2.349 (-13.6%)
HiSTOR0.666 / 0.9040.876 / 1.1500.845 / 1.158

Presentation of the ablation study.

The direct benefit of combining the two encoders is shown under Q3, and the setup shared by all ablation variants is given under Q4.

AuthorsOpenReview ↗

Reviewer2_part3of4

Response to Reviewer 2 (3/4)

Q1. Identifiability of the causal estimands and uncertainty quantification.

Under Eq. (3) the one-step estimand equals E

$Y_{t}$ \| $X_{t}$ = x, $H_{t}$ = h

, and the multi-step estimand is given by the longitudinal g-formula. We will add this as Proposition 1, including the representation-level statement: if (z0z_0, z1z_1) is prognostic for HtH_{t}, ignorability given HtH_{t} carries over to (z0z_0, z1z_1) (Hansen, 2008). On uncertainty: like CRN, CT, Causal CPC and IGC-Net, HiSTOR is a point estimator, and calibrated intervals are orthogonal to the representation question studied here. The paper reports mean ±\pm SD over five seeds for every setting and paired seed-level comparisons for CancerSim (Appendix G).

Q2. Does the estimation target the interventional estimand?

The argument does not rest on the loss alone.

aa One step: the population minimizer of the factual squared loss is E

$Y_{t}$ \| $X_{t}$, $H_{t}$

, which equals the interventional mean under Eq. (3).

bb Multiple steps: the decoder is conditioned on the planned treatment sequence and on the representation of the history at the forecast origin, and predicts all horizons directly instead of chaining one-step predictions. This removes the compounding error of autoregressive rollouts: on the same frozen backbone at dhdd_{\mathrm{hd}} = 2000, autoregressive decoding raises the five-horizon error by 4.8% (Static) and 5.5% (Temporal), and by 7-12% at the intermediate horizons (τ\tau = 3, 4 in Static and τ\tau = 2, 3 in Temporal), over five paired seeds. We will state in Section 4.5 the condition under which the plan-conditional mean coincides with the g-formula target.

dhdd_{\mathrm{hd}} = 2000Decoderτ\tau = 1τ\tau = 2τ\tau = 3τ\tau = 4τ\tau = 5Mean
CVSim StaticDirect (HiSTOR)0.9811.0491.1241.2261.3681.150
CVSim StaticAutoregressive0.975 (-0.6%)1.097 (+4.6%)1.206 (+7.3%)1.325 (+8.0%)1.421 (+3.8%*)1.205 (+4.8%)
CVSim TemporalDirect (HiSTOR)0.9751.0431.1381.2401.3961.158
CVSim TemporalAutoregressive0.927 (-4.9%)1.171 (+12.3%)1.269 (+11.5%)1.349 (+8.7%)1.395 (-0.1%)1.222 (+5.5%)

Both decoders are trained on the same frozen backbone, so the one-step error is identical (0.876 and 0.845).

cc Every multi-step number is computed against the simulator's true counterfactual outcome under the fixed plan, so any deviation from the interventional target is already part of the reported error, and HiSTOR has the lowest five-horizon error in all 23 simulation settings. As a further check we reweighted the decoder loss with stabilized, truncated inverse propensities from HiSTOR's own treatment head. This does not lower the error: the five-horizon NRMSE changes by +4.3% on CancerSim γ\gamma = 4 and by +2.2% and -1.9% on CVSim dhdd_{\mathrm{hd}} = 2000 (Static, Temporal), the latter two within the paired intervals. On these benchmarks the plan-conditional regression is therefore not limited by confounding of future treatments, and we keep the unweighted objective.

Five-horizon NRMSEUnweighted (HiSTOR)IPW-weighted decoderChange
CancerSim γ\gamma = 40.9040.943+4.3%*
CVSim Static, dhdd_{\mathrm{hd}} = 20001.1501.175+2.2%
CVSim Temporal, dhdd_{\mathrm{hd}} = 20001.1581.137-1.9%

The weights are stabilized inverse propensities truncated at the 1st and 99th percentiles, and both decoders use the same frozen backbone. On CancerSim the increase is concentrated at the longest horizons (+6.5%* at τ\tau = 5 and +20.8%* at τ\tau = 6), where the cumulative weights are most variable.

Reviewer_GfZBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies treatment-conditioned potential-outcome prediction from longitudinal observational data when the per-time-step covariates are high-dimensional. The authors propose HiSTOR, which encodes the current covariates with a separate encoder (with reconstruction and WGAN-GP distribution-matching regularizers) and the preceding trajectory with a causally masked Transformer. The fused representation is mapped into a partitioned latent space in which z0 feeds both the outcome head and an auxiliary treatment head, z1 feeds only the outcome head, z2 feeds only the treatment head, and z3 is trained by residual reconstruction. Multi-step predictions are produced either autoregressively or with a direct decoder (a GRU over the treatment plan with one head per horizon). The claimed contributions are a structured representation framework that organizes state and history by their roles in the treatment and outcome mechanisms, an extension of CausalEGM-style structured latent adjustment from the cross-sectional to the longitudinal setting, and the lowest mean NRMSE on CancerSim (γ = 0 to 4) and CVSim (up to 2000 added covariates), plus a semi-synthetic MIMIC-III study in the appendix.

Critical strengths
  1. The problem addresses a real gap. Most neural longitudinal methods (CRN, CT, Causal CPC, IGC-Net) operate on low-dimensional covariates, and the paper explicitly asks how high-dimensional per-time-step measurements should be compressed for counterfactual prediction. The direction itself is worthwhile.

  2. The experiments are broad and the protocol is transparent, so the results are auditable. The study covers 5 confounding levels and 5 horizons on CancerSim and 9 dimensions, 2 modes and 6 targets on CVSim. It reports sample standard deviations over 5 seeds, gives full per-horizon tables (Appendices G and H), and includes seed-level paired comparisons for CancerSim (Tables 15 and 16). Forecast boundaries, per-baseline handling of future covariates (Table 8), metric definitions (Appendix E), and training budgets (Table 9) are all documented, which is unusual in this area.

Critical weaknesses
  1. The structured partition, which is the central contribution, is never ablated or validated. The ablation in Table 4 removes the covariate bottleneck, the distribution matching, the reconstruction and the direct decoding, but it never removes the partition itself, that is, a single entangled latent of equal width with or without the auxiliary treatment head. No constraint enforces the claimed roles: there is no orthogonality, independence or mutual-information penalty as in DR-CFR, TEDVAE or Chu et al. (2022). Nothing prevents z1 from encoding treatment-predictive information, and the treatment head's weight is only λ_X = 0.01 in all CVSim experiments, so the supervision meant to shape z2 and z0 is almost switched off in the high-dimensional setting. In addition, the direct decoder used for all main multi-step results consumes all four branches, including the supposedly irrelevant z3. As it stands, the gains could come from architecture, capacity or input engineering rather than from the partition. An unpartitioned latent of equal width, together with a λ_X = 0 variant and a simple probing analysis (how well treatment can be predicted from z1 and outcome from z2), would directly test contributions one and two.

  2. The experiments do not demonstrate a benefit from high-dimensional compression. In CVSim, every added covariate is a noisy linear measurement of the 3-dimensional patient factor q_i, which is already given to all models as a static input, so the added dimensions carry no new information and are pure nuisance features; the benchmark tests robustness to redundant noise, not the ability to extract confounding information from high-dimensional measurements. Even on this benchmark, HiSTOR's own error grows with dimension: the static one-step error rises from 0.724 to 0.882 (+22%) and the five-horizon mean from 1.000 to 1.197 (+20%), and IGC-Net is better at one-step for d_hd = 10 to 100. The one-step drop from d_hd = 100 to 200 coincides with a switch of the projection network from a sinusoidal-feature MLP to a plain MLP (Appendix B), and the basis for this dimension-dependent choice is not explained. In CancerSim, the covariate input is in fact W_t = [Y_{t-1}, S] with d_W = 2, so the "bottleneck" is an expansion from 2 to 128 dimensions followed by a projection to 16, and the ablation finding that the bottleneck is the most influential component holds only on low-dimensional data and cannot support a high-dimensional-compression claim; no ablation is reported on CVSim. On the only dataset with realistic clinical covariates (MIMIC-III, Appendix I, Table 45), HiSTOR's one-step error is 34% higher than CT's (0.1701 vs. 0.1269), and its multi-step advantage is only 0.3% and 0.2% at τ = 2 and 4 with no standard deviations, so the conclusion's claim of the lowest error in every evaluated setting should be qualified.

Questions for authors
  1. Could you add an ablation that replaces the four-branch partition with a single latent of equal width (dZ = 160), with and without the auxiliary treatment head (λ_X = 0), on CancerSim γ = 4 and CVSim at d_hd = 2000? Does the partition itself help?

  2. How were λ_X (0.01 vs. 1), λ_roll and the d_hd-dependent projection branch selected? Were any of these choices informed by test or counterfactual metrics? What happens on CVSim if the sinusoidal branch is used for d_hd of 200 or more, or the plain MLP for d_hd of 100 or less?

7 comments
AuthorsOpenReview ↗

Reviewer1_part1of6

Response to Reviewer 1 (1/6)

We thank the reviewer for the careful reading. We clarify each point below. All new experiments reported below use the submitted model, and the main result is unchanged by them: HiSTOR has the lowest five-horizon error in all 23 simulation settings of the paper (18 CVSim, 5 CancerSim). Unless stated otherwise, every new number is NRMSE (%) averaged over five seeds, every relative change is paired by seed against the submitted model trained in the same batch of runs, and * marks a change whose 95% paired t-interval excludes zero. "One-step" is the one-step-ahead error and "five-horizon" is the mean over the five multi-step horizons, as in the paper. In these reruns the submitted model obtains 0.666 / 0.904 on CancerSim γ\gamma = 4, 0.876 / 1.150 on CVSim Static dhdd_{\mathrm{hd}} = 2000 and 0.845 / 1.158 on CVSim Temporal dhdd_{\mathrm{hd}} = 2000 (one-step / five-horizon), against 0.670 / 0.911, 0.882 / 1.197 and 0.845 / 1.167 in the paper's tables.

W1. The structured partition is never ablated or validated.

Unpartitioned latent of equal width.

The four slots are a structural prior on what each head reads: the outcome head sees (z0z_0, z1z_1), the treatment head sees (z0z_0, z2z_2), and z3z_3 is reserved for reconstruction. They are separated by this routing. We tested the structure directly with an equal-width model in which every head reads the whole latent, in eight data settings (three seeds each, paired), including new CVSim settings that contain treatment-only and outcome-only covariates. With the paper's loss weights the two models are statistically indistinguishable in every setting (paired 95% intervals include zero), so the partition carries no accuracy cost. It becomes useful when the treatment objective is strong (λX\lambda_{X} = 1): on CVSim Static with latent redundant covariates (dhdd_{\mathrm{hd}} = 2000, the patient factor is withheld from all models), the partitioned model is 10.0% better over five horizons (NRMSE 1.004 against 1.115, paired 95% interval of the difference [-0.155, -0.068]) and 4.7% better one-step (interval [-0.050, -0.024]) than the unpartitioned one, and its one-step error is lower in all seven CVSim settings (by 1.5-4.7%, significant in two). This is consistent with the purpose of the routing, which keeps the treatment-head gradient out of the outcome-specific slot z1z_1. The partition therefore costs nothing in accuracy and protects the outcome representation when the treatment objective is strong. The table of this comparison at the submitted loss weights, with and without the treatment head, is given under Q1.

No constraint enforces the roles of the branches.

No orthogonality, independence or mutual-information penalty is used, and this is deliberate. Under sequential ignorability, valid adjustment requires that the representation read by the outcome head be sufficient for the history with respect to the outcome (a prognostic score, Hansen, 2008). This is a condition on what (z0z_0, z1z_1) contain and does not involve independence from z2z_2. Independence between latent blocks is not identifiable without further inductive bias (Locatello et al., 2019), and penalties that enforce invariance or balance make a representation non-invertible and can discard information that adjustment needs (Johansson et al., 2019, Melnychuk et al., 2024). HiSTOR therefore assigns the supervision by routing: the outcome loss can only shape (z0z_0, z1z_1), the treatment loss only (z0z_0, z2z_2), and the residual reconstruction only z3z_3, while the reconstruction objective keeps the latent information-preserving.

λX\lambda_{X} = 0.01 on CVSim.

λX\lambda_{X} = 0.01 on CVSim matches gradient scales rather than switching the treatment head off: the standardized outcome MSE is about 0.02 and the three-class cross-entropy about 0.7-1.1, so λX\lambda_{X} = 1 would let the treatment term outweigh the outcome term by a factor of about 50. The sensitivity study confirms this at dhdd_{\mathrm{hd}} = 2000 (Static): with λX\lambda_{X} = 1 the error rises by 7.7% (one-step) and 11.3% (five-horizon), and by 17% at τ\tau = 1 (paired 95% interval excludes 0), whereas λX\lambda_{X} = 0.1 and λX\lambda_{X} = 0 leave it within ±2.4%. The submitted value λX\lambda_{X} = 0.01 gives the lowest one-step error of the five values and a five-horizon error within 0.7% of the lowest.

λX\lambda_{X} (CVSim Static, dhdd_{\mathrm{hd}} = 2000)One-stepFive-horizon
00.897 (+2.4%)1.164 (+1.3%)
0.0010.883 (+0.8%)1.238 (+7.6%)
0.01 (submitted)0.8761.150
0.10.884 (+0.9%)1.161 (+0.7%)
10.943 (+7.7%)1.279 (+11.3%)
AuthorsOpenReview ↗

Reviewer1_part2of6

Response to Reviewer 1 (2/6)

(W1 continued)

The direct decoder reads z3z_3.

On z3z_3 in the decoder: we retrained the decoder on the same frozen backbone with its input restricted to

$z_0$, $z_1$, $z_2$

(120 instead of 160 dimensions). The five-horizon error rises from 0.904 to 0.911 (+0.7%) on CancerSim (γ\gamma = 4), from 1.150 to 1.201 (+4.5%) on CVSim Static and from 1.158 to 1.176 (+1.5%) on CVSim Temporal (dhdd_{\mathrm{hd}} = 2000). The one-step error is unchanged by construction, since the backbone is the same. None of these differences is significant, so reading z3z_3 does not hurt the decoder and tends to help at longer horizons on CVSim Static (+10.7% at τ\tau = 5 without it).

W2. The experiments do not demonstrate a benefit from high-dimensional compression.

The added CVSim covariates are redundant measurements of the observed patient factor.

This is the intended design, stated in Section 5.3 ("redundant high-dimensional measurements") and Appendix C. CVSim mirrors the structure of clinical records, in which thousands of measured variables are noisy, correlated readouts of a low-dimensional physiological state. The benchmark isolates the difficulty specific to high-dimensional longitudinal data: the signal dimension is fixed while the input width grows from 7 to 2007. The baselines show that this is not a trivial test. At dhdd_{\mathrm{hd}} = 2000 (Static) the five-horizon NRMSE is 2.22 for CT, 3.40 for RMSN, 4.47 for CTD-NKO and 1.67 for Causal CPC, against 1.20 for HiSTOR, although every model receives the same information.

To also cover the regime in which the added dimensions are the only source of the patient factor, we ran a latent variant in which qiq_{i} is withheld from all models (Static, dhdd_{\mathrm{hd}} ∈\in {0, 10, 100, 500, 2000}, five seeds, with CT, Causal CPC and IGC-Net). HiSTOR does not depend on being given the patient factor: without it, its error at dhdd_{\mathrm{hd}} = 2000 is 0.883 (one-step) and 1.218 (five-horizon), against 0.876 and 1.150 with it, and it stays in a narrow range over all dimensions (one-step 0.75-0.88, five-horizon 1.10-1.30). Over five horizons HiSTOR is best at four of the five dimensions. At dhdd_{\mathrm{hd}} = 100 Causal CPC is 2.6% lower, within the paired interval. At dhdd_{\mathrm{hd}} = 2000 the baselines are 36% (Causal CPC), 74% (CT) and 100% (IGC-Net) above HiSTOR over five horizons and 42%, 34% and 8% above it at one step.

dhdd_{\mathrm{hd}} (latent, Static)HiSTORCausal CPCCTIGC-Net
00.749 / 1.1131.019 / 1.7171.285* / 2.270*0.819 / 2.389*
100.818 / 1.1300.924 / 1.3911.301 / 2.090*0.812 / 2.534*
1000.879 / 1.3020.976 / 1.2691.106* / 2.125*0.813 / 2.539*
5000.808 / 1.1000.992 / 1.3351.089* / 2.209*0.839 / 2.358*
20000.883 / 1.2181.255 / 1.655*1.180* / 2.118*0.914 / 2.415

Entries are one-step / five-horizon NRMSE. * marks a paired difference from HiSTOR whose 95% interval excludes zero (five seeds, four for IGC-Net at dhdd_{\mathrm{hd}} = 2000).

AuthorsOpenReview ↗

Reviewer1_part3of6

Response to Reviewer 1 (3/6)

(W2 continued)

HiSTOR's error increases with dimension. IGC-Net is better at one step for dhdd_{\mathrm{hd}} = 10-100.

The input width grows by a factor of 287 while HiSTOR's five-horizon error grows by 20% (Static, 1.000 →\to 1.197) and 19% (Temporal, 0.983 →\to 1.167). Most of the increase occurs below dhdd_{\mathrm{hd}} = 100. From dhdd_{\mathrm{hd}} = 200 to 2000, a tenfold increase, the error rises by 12% in both settings.

Over five horizons HiSTOR has the lowest error in all 18 CVSim settings. Its margin over the best baseline is 22-28% for dhdd_{\mathrm{hd}} ≥\geq 200 in the Static setting (28.4% at dhdd_{\mathrm{hd}} = 2000: 1.197 vs. 1.671) and 11-38% in the Temporal setting. The baselines that are competitive at low dimension degrade far more over the same range: Causal CPC +33%, RMSN +142%, CTD-NKO +244% (Static). CT and IGC-Net change little, but their five-horizon error is about twice HiSTOR's at every dimension (CT 1.81-2.36, IGC-Net 2.25-2.49, HiSTOR 1.00-1.24).

At one step HiSTOR is best at dhdd_{\mathrm{hd}} = 0 and at every dhdd_{\mathrm{hd}} ≥\geq 200 (10 of 18 settings, 7.5% and 10.2% ahead at dhdd_{\mathrm{hd}} = 2000), and IGC-Net is 2-11% ahead for dhdd_{\mathrm{hd}} = 10-100. The reviewer is right that the submitted text did not make this split clear enough, and we will state it explicitly in Section 5.3. HiSTOR is the only method that ranks first or second on both tasks in all 18 settings. The projection-swap control reported under Q2 ties the low-dimensional one-step gap to the Fourier branch that the submitted rule uses there: with the plain MLP, HiSTOR's one-step error at dhdd_{\mathrm{hd}} = 10, 50 and 100 is 0.740, 0.801 and 0.798 (Static) and 0.739, 0.778 and 0.769 (Temporal), against 0.795, 0.794, 0.829 and 0.794, 0.810, 0.821 for IGC-Net in the paper's tables.

The change between dhdd_{\mathrm{hd}} = 100 and 200 coincides with the projection switch.

The projection follows one fixed architectural rule, and the swap experiment the reviewer asks for is reported under Q2. Its result is that the change between dhdd_{\mathrm{hd}} = 100 and 200 is the transition from the weaker to the stronger branch and not an effect of dimension: with the plain MLP at every dimension, the Static five-horizon error is 0.99, 1.10, 1.04, 1.11 and 1.15 for dhdd_{\mathrm{hd}} = 0, 10, 50, 100 and 200, with no discontinuity.

CancerSim has dWd_{W} = 2, so its ablation does not support high-dimensional compression.

The compression in HiSTOR targets the covariate stream, not the outcome value: the bottleneck E(⋅\cdot) is applied to the covariates observed at the current step, and the 128-dimensional layer before it is a learned embedding of the raw input, so the current covariates enter the latent through the 16-dimensional code. We agree that our wording around the CancerSim ablation was imprecise on this point. On CancerSim, where dWd_{W} = 2, the ablation tests the mechanism (encode the current state separately, then fuse it with history) on the benchmark shared by all prior work. The high-dimensional claim rests on CVSim, where we tested the role of the current state directly at dhdd_{\mathrm{hd}} = 2000. Removing the current-state path, so that the model reads the history tokens only, raises the one-step error by 27.5% (Static) and 30.8% (Temporal) and the five-horizon error by 15.8% and 12.4% (the latter with a paired 95% interval that excludes 0). A separate, compressed representation of the current high-dimensional state is therefore what the model relies on at high dimension.

dhdd_{\mathrm{hd}} = 2000HiSTOR (one-step / five-horizon)Without the current-state pathChange
CVSim Static0.876 / 1.1501.117 / 1.331+27.5% / +15.8%
CVSim Temporal0.845 / 1.1581.105 / 1.302+30.8% / +12.4%*

The degradation grows with the horizon: on CVSim Temporal it is +7.0%, +10.3%, +12.1%*, +14.8%* and +16.0%* at τ\tau = 1, …, 5, and on CVSim Static +10.8%, +9.9%, +15.5%, +19.3% and +21.0%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.