- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces a special attention mechanism for modelling structural causal models. The key innovation is a construction that preserves parent restrictions at arbitrary depth and supports explicit counterfactual semantics. This is achieved by introducing separate computation and value tokens. The suggested architecture is evaluated in a range of benchmarks and compared against a regular transformer and a random-sparse transformer.
Critical strengths
The construction and augmentation of the attention design via computational and value tokens is elegant and supports counterfactual semantics.
Critical weaknesses
I see the following weaknesses:
-
there has been many alternative neural architectures that have been proposed for structural causal models, e.g., under the rubrik of 'neural causal models', 'DAG-aware transformers', 'deep SCMs', 'causal normalizing flows', etc. The paper should compare and benchmark specifically against those, as the current baselines (regular transformer and random-sparse transformers) do not have access to the DAG.
-
the paper should isolate the benefit of the architectural modification. Compared to the earlier deep structural causal models that directly parametrize the structural equations, the current solution modifies the attention layers by introducing the DAG and the computational/value tokens. These changes should be ablated, and the performance gains compared to causual baselines is unclear at the moment.
-
Algorithm 2 for counterfactual inference starts by drawing from the posterior P_\theta(U|Z=z). It is unclear whether these posterior draws can be computed efficiently, and the suggested rejection sampling scheme seems to have extremely small acceptance probability in general. It is therefore unclear whether counterfactual inference is computationally feasible with the suggested architecture.
Overall, while I enjoyed reading the paper, the main question that remained unaswered was 'what benefit comes specifically from using a transformer/multi-layered attention, beyond the supplied structural equations?'.
Questions for authors
Please respond to the concerns raised in weaknesses.
In addition, I was unsure about the foundation model example. How is the underlying DAG of the structural causal model chosen in this example?
6 comments
Author Response [2/4]
To still provide an empirical comparison with existing methods that use causal structure, we adapted NCM, DAG-aware Transformer, DeepSCM, Causal Normalizing Flows to our setting. To make the comparison possible, we focus on the query over the target alone instead of the full trajectory, i.e. . Please refer to the App. H for the full details about the adaptation of each baseline.
| Method | MSE ↓ | Error reduction vs. Naive ↑ | Pearson r ↑ | Directional accuracy ↑ |
|---|---|---|---|---|
| CaLM | 0.0856 | +3.7% | 0.49 | 73.3% |
| DAG-aware Transformer (best variant using G-formula, rest in Fig. 18) | 0.1071 | −20.5% | 0.21 | 73.3% |
| NCM | 0.0994 | −11.7% | 0.30 | 69.2% |
| Causal NF | 0.0980 | −10.2% | 0.30 | 70.8% |
| DeepSCM | 0.1081 | −21.6% | 0.03 | 73.3% |
CaLM outperforms all adapted baselines and it achieves the lowest counterfactual MSE and the highest Pearson correlation on Tier M. The lower MSE means that CaLM's predicted counterfactual outcomes are closer to the ground truth, and the higher Pearson correlation means that its predictions better track how the ground truth outcome varies across patients and interventions. Moreover, CaLM is the only method that improves over the Naive Transformer; none of the adapted baselines reduces the error relative to Naive. Please refer to the App. H and Fig. 18 for the full results.
Response to W2.
the paper should isolate the benefit of the architectural modification… What benefit comes specifically from using a transformer/multi-layered attention, beyond the supplied structural equations?
We added studies of three architectural design choices in Appendix I:
- Causal-diagram masking versus dense left-to-right masking.
- Computation/readout/value (CRV) blocks versus one token per variable with parent-only masking.
- A shared Transformer versus separate feedforward networks for the structural mechanisms.
The first two studies test whether the architecture correctly enforces the causal structure. The third compares alternative parameterizations of the mechanisms.
Setup. For (1) and (2), we conduct a controlled ablation on three causal graphs (Fig. 19a–c). Models are trained on observational data and evaluated on identifiable interventional queries. Identifiability lets us assess whether the models recover the correct queries without ambiguity from non-identifiability.
Ablation arms. We compare four variants:
- CaLM: graph-based mask with CRV blocks.
- CRV only: dense left-to-right mask with CRV blocks.
- Graph mask only: parent-only mask with one token per variable.
- Naive: dense left-to-right mask with one token per variable.
The Transformer backbone, observational training data, and optimizer are held fixed across these arms.
Metrics. We measure observational fit, interventional query error, and non-parent leakage. Leakage measures how much a variable’s predicted distribution changes when non-parent values vary while its parent values remain fixed. This directly tests the information-leakage mechanisms discussed in Section 3 and Appendix I.1.
Results. The results are illustrated in the following table. All four variants achieve similar observational fit, but only CaLM has zero non-parent leakage across all tested variable orders and achieves the lowest query error. This suggests the importance of both (1) and (2) in architecture design.
| Variant | Attention mask | CRV blocks | Observational error ↓ | Interventional error ↓ | Non-parent leakage ↓ |
|---|---|---|---|---|---|
| CaLM | Graph-based | Yes | 0.008 | 0.008 | 0.000 |
| CRV only | Dense left-to-right | Yes | 0.008 | 0.154 | 0.299 |
| Graph mask only | Parent-only | No | 0.008 | 0.165 | 0.389 |
| Naive | Dense left-to-right | No | 0.008 | 0.168 | 0.823 |
For (3), we additionally compare CaLM with an MLP-based MLE-NCM using the same Sepsis causal graph, reusing the baseline experiment in Appendix H. Both methods can encode the supplied causal structure. At Tier M, CaLM achieves lower counterfactual MSE (0.0856 versus 0.0994) and higher Pearson correlation (0.49 versus 0.30) than the evaluated MLE-NCM with likelihood weighting.
Full experimental setups, analysis, and results are provided in Appendices I.1 and I.2.
Thus, the architectural benefits from using CaLM, and thus a transformer based architecture is highlighted by the results presented in App. I. Furthermore, using transformers, also allows us to fine-tune foundation models, as in App. D.8.
Author Response [3/4]
Response to W3.
Algorithm 2 … It is therefore unclear whether counterfactual inference is computationally feasible with the suggested architecture.
We acknowledge the lack of context and elaboration about the posterior sampling procedure, and we appreciate the reviewer for noticing the gap in the presentation; we included more elaboration in the paper (highlighted in blue), and we shall also release the code upon acceptance.
As mentioned in the paper, rejection-sampling and posterior modeling are conventional choices for the posterior step, and in CaLM we used the same transformer architecture as a posterior sampler. In particular, the unobserved variables are placed in the context, preceding the observable variables , which are followed by a posterior sampler restricted to only attending . We train for conventional next-token prediction loss, to reconstruct from . As we train the model to fit the observational data, the segment learns a posterior sampler with negligible cost. The sampler adds no separate network or training stage. The algorithms in the appendix digressed the specifics of implementation for sake of generality, and to maintain coherence with the notation/conventions introduced in the theory section.
Below, we provide some empirical feasibility.
Number of samples and accuracy. We use for sampling in the abduction steps. We can therefore compare the sampler with the exact model posterior. Results are over 5 training seeds, with 1000 repeated runs of the estimator per query. Both metrics are computed per query and averaged with equal weight over queries. The sampler’s approximation error is as follows: , and nats. Abduction still lowers the counterfactual error. At Tier S it falls from to (95% CI of the gain: ). The error decreases monotonically in and approaches the limit from above. The remaining error comes from , not Monte Carlo noise. The induced error is bounded by , and is much smaller when the misplaced mass falls on values of with the same outcome. Empirically it is of and never exceeds it. The sampler is approximate, but it shifts mass in the direction that matters for the query, which is enough to cut the no-abduction error by about two thirds at Tier S.
Computational cost. Alg. 1 trains the auxiliary variables in the same teacher-forced forward pass as the mechanisms, so no separate network or training stage is needed. In Alg. 2, all draws run as one batch. Drawing from takes one forward pass per exogenous variable. Prediction takes one pass per decoded variable, and the clamped intervention takes none. The number of passes therefore does not depend on ; only sets the batch size. Because the rollout under a fixed is deterministic, repeated draws need to be decoded only once.
At N = 64 on one H100:
- Sepsis Tier M: 3.4 s per query and 881 MiB. Abduction takes 0.12 s of this (4%).
- Mistral-7B / Llama-3-8B: 6.0 s / 4.9 s per query and 26 / 31 GiB. Abduction takes 1.0 s / 0.8 s of this (17%).
Author Response [1/4]
We are grateful to the reviewer for their detailed comments.
Response to W1.
We address this point in two steps: we first discuss why existing baselines that use causal structure are not directly applicable to our setting, and then provide an empirical comparison with versions of these baselines adapted for counterfactual simulation. We add all these discussions in the App. H.
Applicability of the DAG baselines
CaLM is designed for large-scale counterfactual trajectory simulation under a general causal diagram. This requires three capabilities at once: (1) counterfactual trajectory simulation with abduction (an L3 query); (2) long sequences whose number of variables is not fixed in advance; and (3) a general causal structure with unobserved confounding. Each of the cited baselines, although it builds on causal assumptions (for example, a given DAG) to estimate causal queries, lacks at least one of these capabilities, so none can be applied directly to the tasks in Sec. 4.
(1) Counterfactual trajectory simulation with abduction (L3 query). Our target is the counterfactual trajectory distribution P(Zx′∣Z=z), which requires abduction of the exogenous variables, P(U∣Z=z), from the factual trajectory and reuse of the same U in the intervened world. In our ICU experiment, for instance, the query is: given a patient's completed ICU record z1:T, what would the full record have been had the clinician given vasopressor dose x′ at hour t instead of the factual one, i.e. P(Z1:T[Xt=x′]∣Z1:T=z1:T) (App. C.4.3). Answering it requires inferring the patient's unobserved frailty UF from the whole record and keeping it fixed under the alternative treatment. DAG-aware Transformer [6] estimates an interventional query on the outcome Y only, not on the trajectory. Specifically, it fits the propensity score and the outcome regression from observational data and plugs them into standard formulas (g-formula, IPW, AIPW) for the ATE or CATE. It has no procedure for counterfactual queries, which require inferring P(U∣Z=z). NCM [3], DeepSCM [4] and Causal Normalizing Flows [5] do support abduction, but they fall short on (2) or (3) below. CaLM performs abduction and generates the full trajectory in one model: P(U∣Z=z) is inferred within the sequence prediction itself, with an attention mask ensuring that the inferred U takes only the factual trajectory as input.
(2) Long sequences, number of variables not fixed in advance. We work in settings where the simulated sequence is long, and its length is not fixed in advance (for example, up to 2,000 tokens in chess). NCM, DeepSCM, Causal Normalizing Flows and DAG-aware Transformer are defined over a fixed set of variables (one network, one flow dimension or one token per variable), which must be specified in advance. In addition, NCM answers conditional counterfactual queries by sampling u from the prior, keeping the draws that reproduce the evidence, i.e. Z(u)=z, and rejecting the others. As the sequence length increases, the acceptance rate decays and a very large number of samples of u is needed.
(3) General structure with unobserved confounding. CaLM takes an arbitrary causal diagram G as input, including bidirected edges for unobserved confounding. Thm. 9 proves that CaLM is sound and expressive for representing any SCM compatible with a given causal diagram. For example, in the driving domain, each driver's actions are confounded by that driver's unobserved persona. In contrast, DeepSCM and Causal Normalizing Flows assume an independent exogenous variable U for each variable, and DAG-aware Transformer assumes unconfoundedness (or valid proxy variables), i.e. that adjusting for the observed covariates removes all confounding between the treatment X and the outcome Y. These assumptions do not hold in our settings. NCM does handle general causal diagrams, but it falls short on (2).