- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies treatment effect estimation and multi-step counterfactual prediction under policy and environment shifts. It proposes DECAF, a two-stage framework that first separates patient history into outcome-related and environment-related subspaces, then freezes the encoder and uses conditional diffusion with contrastive regularization to predict future states in the outcome-related subspace. Experiments on a tumor growth simulator, semi-synthetic MIMIC-III, and real MIMIC-IV show lower average errors than baselines in most cross-environment prediction tasks.
Critical strengths
The paper has a clear motivation and combines environment separation with multi-step counterfactual prediction. It separates outcome-related and environment-related information, then predicts future states in the outcome-related subspace. This design may reduce the accumulation of environment-specific bias over multiple steps.
The experiments are systematic, with several baselines on simulated, semi-synthetic, and real clinical data. Ablation studies examine different generative models and loss terms. The authors also clearly state the theoretical assumptions and the lack of counterfactual ground truth in real data, helping readers understand the limits of the results.
Critical weaknesses
The first stage uses outcome supervision only for one-step prediction, then freezes the encoder. It is unclear whether the learned representation retains the information needed for longer-term predictions. Since the environment and patient condition may be related, removing environment information could also remove useful clinical information that one-step supervision may not preserve.
Evidence for multi-step counterfactual prediction remains limited, mainly coming from the tumor simulator. Semi-synthetic MIMIC-III evaluates only one-step treatment effects, while real MIMIC-IV evaluates only factual prediction. Some long-horizon gains are also small compared with the variation across random seeds, so their stability needs further validation.
Questions for authors
-
The first stage uses outcome supervision only for one-step prediction. What evidence shows that the frozen representation retains the information needed for multi-step prediction?
-
Could the model match positive pairs mainly through their shared environment vector, without sufficiently constraining the generated counterfactual states?
-
How do DECAF and the baselines differ in their use of environment labels and tuning budgets? How can DECAF be used when training labels for environments are unavailable, or when the environment changes within a patient’s trajectory?
-
Which data-generating mechanisms and parameters differ across the eight semi-synthetic MIMIC-III environments? Does this setup satisfy the theory’s assumptions about stable mechanisms across environments?
6 comments
Question 2. Matching through the shared environment vector
Response: The generated state comes from the diffusion sampler given , and enters only the projection heads, not the sampler. If the pair loss acted only through , removing the loss would not change the rollout, but it does (Table 10: 1.525 to 1.923, 1.880 to 1.997).
We first retrained Stage 2 without in either projection head (variant B in our reply to Reviewer oC7g). The error increased from 1.525 to 1.594 at and from 1.880 to 1.948 at . We also tested the full model by shuffling the generated state, , or both across patients while keeping the projection heads fixed. The retrieval accuracies are shown below:
| Generated side | Retrieval accuracy (%) |
|---|---|
| Original | 54.3 |
| Generated state permuted | 7.8 |
| permuted | 35.6 |
| Both permuted | 1.6 |
Most of the matching comes from the generated state; acts as context.
Question 3. Environment labels; tuning; missing or changing labels
Response: All methods are trained in the same setting. DeCaf uses the environment label only in the Stage-1 losses (Eq. 7), one per trajectory (policy in Tumor Growth, SCM environment in MIMIC-III, care unit in MIMIC-IV); Stage 2 and inference do not use it. As this input is specific to DeCaf, we checked whether it explains the improvement: variant A (reply to Reviewer oC7g) uses the same label through the adversarial objective, without the branch, and is worse at every horizon (1.492 / 1.727 / 2.083 vs 1.371 / 1.525 / 1.880). The label alone does not explain the improvement.
On tuning efforts, we used the same configuration for DeCaf across all three benchmarks, without dataset-specific tuning (App. A.4), the baselines use the defaults of Melnychuk et al. (2022) (App. F), and every method is early-stopped on the same source validation split, with no OOD data used for selection (App. D.1).
We use one environment label per trajectory during training, such as the care unit in our MIMIC-IV experiments. If labels are unavailable, one option would be to infer environments from the data. If the environment changes within a trajectory, the classifiers in Eq. (7) could use a different label at each time step. Both settings are beyond the scope of this paper, but we appreciate the reviewer raising them. They are important in real world settings, for transfer across hospitals and for patients who move between units. We will discuss them in Sec. 7 as limitations and future work.
Question 4. MIMIC-III environments; stable-mechanism assumption
Response: Thank you for the careful reading. The eight environments use the same SCM equations and observed covariates; they differ in patients and in the parameter values (autoregressive and covariate weights, treatment-effect intercept and slope, noise scale).
Theorem 5 states a sufficient condition for a shared predictor on to be invariant across environments. In the Tumor Growth dataset, environments differ only in the treatment-assignment policy, while the simulator's response mechanism remains fixed (App. D.2). The semi-synthetic MIMIC-III dataset tests a different setting: the held-out environments 4–7 use different SCM parameter values, introducing changes in the outcome mechanism itself. DeCaf still has the lowest OOD √PEHE there (0.314 vs 0.434, Table 3). DeCaf improves under both shifts: a policy shift with a fixed response mechanism, and a change in the outcome mechanism itself.
Weakness 1. One-step supervision; loss of information
Please see our response to Q1.
Weakness 2. Multi-step evidence
Response: We extended the semi-synthetic MIMIC-III benchmark to treatment plans of length under the counterfactual protocol (App. D.2): the SCM is iterated under the plan with the observed covariates held fixed, Stage 2 is trained on the source environments, and we report (mean ± SD, three seeds).
| , MIMIC-III | |||
|---|---|---|---|
| CDM | 0.452 ± 0.023 | 0.729 ± 0.035 | 1.495 ± 0.052 |
| DR-CRN | 0.535 ± 0.024 | 0.795 ± 0.046 | 1.764 ± 0.083 |
| G-Transformer | 0.492 ± 0.062 | 0.824 ± 0.058 | 1.964 ± 0.091 |
| DeCaf | 0.394 ± 0.031 | 0.712 ± 0.041 | 1.376 ± 0.042 |
DeCaf has the lowest mean error at all three reported horizons. The difference from CDM is small at and larger at , where DeCaf reduces the mean error by 8.0%. These results provide additional multi-step counterfactual evidence beyond the tumor simulator.
We thank the reviewer and would be glad to discuss any remaining concerns
Dear Reviewer,
Thank you again for taking the time to review our paper and share your feedback. We have included additional experiments and analyses in our response. When you have a chance, we would appreciate your thoughts and would be happy to discuss any remaining questions.
We thank the reviewer for the careful and constructive review.
Response: Thank you for raising this concern. Multi-step prediction (Eq. 3) applies the one-step transition K under each treatment and then the one-step head mt,a(ztc) of Theorem 1:
E[yt+τaˉ∣Ht]=∫mt+τ−1,at+τ−1(zt+τ−1)s=t∏t+τ−2Ks,as(dzs+1∣zs),zt=ztc
This needs two conditions: a correct one-step head, and ztc carrying everything in Ht that matters for the next state (transition sufficiency, Eq. 2). One-step supervision does not guarantee the second, so Section 5 states it separately. When both hold, with head error ϵy, head bound B and per-step transition error ϵK (total variation), the rollout error satisfies
∣μτ−μ^τ∣≤ϵy+2B(τ−1)ϵK
so it grows at most linearly with the horizon; we will add the bound and proof to App. H. Whether training meets these conditions is an empirical question, which the experiments below address.
First, we examined whether one-step supervision is the limiting factor. If one-step supervision discarded information needed for rollout, multi-step targets should improve the rollout. We tested two variants under the Table 10 protocol (Tumor Growth nRMSE, four-policy mean over three seeds; Δ: per-seed difference to DeCaf, mean ± SD): in (i), Stage 1 gets extra factual heads that predict yt+τ for τ≤5 under the realized treatments, encoder frozen; in (ii), the encoder is fine-tuned with Stage 2, keeping the Stage-1 outcome loss.
Fine-tuning the encoder with Stage 2 (without the separation losses) helps a little at one step but hurts at three and five. Multi-step heads cost a little at one step and help a little at five. The frozen one-step encoder retains most of what the rollout needs; we will include the multi-step-head variant in the revision.
Second, we examined how much the history knows about the future beyond ztc. If ztc carries everything in Ht that matters for the future, the rollout error cannot be predicted from Ht any better than from ztc itself. For each τ we predict the rollout error yt+τ−μ^τ(ztc) on factual trajectories from ztc, from Ht, and from zte (R2 of the residual, means over three seeds). The zc row measures fitting error of the model itself; the excess of the Ht row over it is the information in the history that ztc misses; the ze row answers the second part of W1.
Using the full history instead of ztc improves residual R2 by only 0.01–0.04 on Tumor Growth and 0.03–0.06 on MIMIC-IV. Most of the predictable residual is fitting error that ztc itself predicts, not information missing from it. The probes using zte alone reach residual R2 of at most 0.03 and 0.06, respectively.
Third, the multi-step results already in the paper point the same way. zc keeps 97.9% of the full-history next-outcome R2 on MIMIC-IV (Table 12); on Tumor Growth the four-policy mean error grows by 0.13–0.19 per step from τ=2 to 5 without compounding (Table 11), and at τ=5 DeCaf has the lowest or near-lowest mean in all four policies (Table 9).
In summary, the frozen encoder performs well in multi-step prediction. Adding multi-step supervision gives only a small improvement, while fine-tuning the encoder does not improve the longer-horizon results.