acceptodds
Under review as a conference paper at ICLR 2027

Mask2Cause: Temporal Causal Discovery Beyond Causality in Mean

Abstract

One approach to discovering causal relationships in multivariate time series is to ask whether a variable’s history improves prediction of another variable. When this improvement is measured solely by reductions in optimal squared error, the criterion misses relationships that affect the target’s conditional variance without changing its conditional mean. We propose Mask2Cause, an end-to-end Transformer framework for causal discovery through conditional mean and variance prediction. Each variable’s history is embedded as a separate token, and a single learnable adjacency matrix, shared across layers and attention heads, controls information exchange between tokens. Gaussian negative log-likelihood and a sparsity penalty on this matrix jointly train the forecasting model and graph. Our population analysis connects the optimal log-loss reduction to causally conditioned directed information, providing an information-theoretic motivation for the training objective. We introduce Mixed Physics, a benchmark with separately controlled mean and variance dependencies, to evaluate recovery of mean-only and variance-only edges. In its pure-variance setting, Mask2Cause achieves 1.000 off-diagonal AUROC, compared with 0.521 for a corresponding squared-error model. Experiments on linear, chaotic, biological, and real-data-derived benchmarks demonstrate competitive graph recovery, while the inferred graphs support downstream tasks such as causal pruning for efficient forecasting and root-cause localization.

Primary area:
causal reasoning
Keywords:
causal discovery, Granger causality, multivariate time series, heteroscedasticity, causality in variance, directed information, masked attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4rUROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mask2Cause, an end-to-end Transformer framework for causal discovery through conditional mean and variance prediction. Its main idea is to capture relationships that affect a variable’s variability but leave its mean unchanged. The experiments cover several benchmarks and downstream tasks, including forecasting and root cause localization.

Critical strengths
  1. The motivation is clear. Looking beyond mean prediction addresses a useful limitation of MSE-based methods.

  2. The experiments are fairly broad, with useful comparisons, ablations, and additional analyses in the appendix.

Critical weaknesses
  1. In the multi-layer encoder, blocking a direct connection does not prevent information from reaching the target through another token. This makes the interpretation of the learned adjacency as direct dependencies less clear.

  2. RHINO also achieves perfect recovery in the pure-variance setting in Tab. 4. A clearer comparison of recovery accuracy and computational cost would help show M2C’s practical advantage.

  3. Several gains in Tab. 3 are small, and may require statistical evidence.

Questions for authors
  1. How does information passing through intermediate tokens affect the interpretation of the learned adjacency in the multi-layer encoder?

  2. Can you summarize the accuracy and computational cost trade-off between M2C and RHINO under the same experimental settings?

  3. Are the small improvements in Tab. 3 consistent across repeated experiments?

2 comments

Q1

This possibility is explicitly discussed in Appendix G. Lines 1109–1110 state: “In a multi-layer setting, a theoretical discrepancy arises between the ideal estimator assumption and the structural mask.” We can therefore interpret the adjacency as a ranking of candidate parents. We discuss what supports that interpretation in our response to Reviewer yjvT, W1.

Q2

Table 4 shows that M2C-NLL matches or exceeds RHINO’s mean off-diagonal AUROC in six of seven settings. On RHINO’s history-dependent Erdős–Rényi benchmark, M2C-NLL leads at N=5,10,40N=5,10,40, with the largest gain at N=5N=5 (0.885 versus 0.798). The two methods are nearly tied at N=20N=20 (0.888 versus 0.890). On Mixed Physics, M2C-NLL leads by 0.030 and 0.023 in the 50:50 and 75:25 regimes, respectively, and both achieve 1.000 in the pure-variance regime.

We measured the wall-clock time for a full training run using the selected settings, including the training duration selected during calibration for M2C-NLL (Appendix~O.3). The timings exclude hyperparameter search.

On Mixed Physics (50:5050{:}50, 75:2575{:}25, and 100:0100{:}0), M2C-NLL took 2.96, 1.42, and 0.83 minutes, respectively, compared with 24.22, 24.70, and 24.60 minutes for RHINO. On RHINO ER (N=5,10,20,40N=5,10,20,40), M2C-NLL took 1.02, 3.88, 0.86, and 12.64 minutes, respectively, compared with 97.75, 42.04, 39.36, and 44.63 minutes for RHINO. Across the seven datasets, M2C-NLL achieved a 3.5–95.8× speedup over RHINO, with a mean of 30.2x.

Table 20 shows the scaling of compute with NN and LL for a single forward pass. RHINO has a smaller footprint at N=10,L=5N=10, L=5, with 0.0323M parameters and 1.80M FLOPs versus 0.1006M and 2.05M for M2C-NLL. However at scale, M2C-NLL is significantly cheaper. At N=2000,L=5N=2000, L=5, M2C-NLL uses 4.10M parameters and 2.51B FLOPs versus 70.66M and 8.28B for RHINO. At N=10,L=2000N=10, L=2000, M2C-NLL uses 0.2283M parameters and 4.60M FLOPs versus 1.0298M and 555.20M for RHINO.

Q3

We repeated the DREAM3 experiments with five training seeds per model and network. The data and hyperparameters were fixed. Only weight initialization, mini-batch order, and dropout varied. AUROC was stable across runs, with standard deviations of 0.001–0.005. The clearest gain was on E. coli-2, where M2C-NLL averaged 0.703±0.0030.703 \pm 0.003 versus 0.6660.666 for the strongest baseline. Its leads on E. coli-1 and Yeast-2 were smaller (0.004 and 0.003), and we do not claim significant gains there. The intended role of DREAM3 is stated in Section 5.2.1, lines 325–326: “DREAM3, which features extremely short trajectories T=21T=21, is purely used to evaluate how the model fares against baselines in highly data-scarce domains.” The new five-seed results in Appendix W support the stability of M2C’s scores in this setting.

Thank you for the detailed rebuttal and additional experiments. While I appreciate the authors' efforts, the rebuttal does not change my overall assessment of the paper. Therefore, I will maintain my original score.

Reviewer_JbqUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Mask2Cause (M2C), a Transformer-based causal discovery framework that learns a shared adjacency matrix through masked attention and jointly optimizes forecasting. It uses Gaussian negative log-likelihood to identify both mean- and variance-driven dependencies, supported by directed-information arguments and experiments on synthetic and real-world benchmarks.

Critical strengths

Joint learning of the causal graph and forecasting model avoids separate post-hoc graph extraction and demonstrates competitive empirical performance.

Critical weaknesses
  1. The directed-information argument applies to optimal conditional predictors but does not establish that the sparsity-regularized Transformer recovers the true causal adjacency matrix.
  2. Mixed Physics is the primary benchmark with explicitly controlled variance-driven ground truth, and stronger comparisons against equally expressive heteroscedastic likelihood-based models are needed to isolate the proposed architecture's contribution.
  3. Gaussian NLL captures changes in conditional mean and variance but cannot identify dependencies expressed exclusively through higher-order moments, limiting the claimed connection to general directed information.
Questions for authors

Q1. Can the authors provide theoretical conditions under which the learned adjacency matrix consistently recovers the directed-information graph?

Q2. Can the authors compare M2C against matched Transformer baselines using the same Gaussian NLL objective to separate architectural gains from likelihood-related gains?

Q3. How does M2C perform when causal dependencies change conditional skewness or other higher-order moments while preserving the mean and variance?

4 comments

Q1

The paper explicitly distinguishes the population prediction result from consistency of the learned graph. Appendix M, line 1701, states: “We prove no consistency or identifiability result for the learned adjacency matrix.” Appendix U.2 makes the same distinction at the end of the corollary, lines 2307–2308: “This identifies the same population edges, not necessarily the same numerical loss reduction as DI, and does not assert recovery by a fitted network.”

Our result establishes when excluding a source increases optimal expected Gaussian NLL. A graph recovery guarantee would require showing that the true graph is identifiable when the gates and predictor are learned jointly, and that gradient descent on the nonconvex objective recovers it as the sample size grows. Our result motivates the forecasting loss. Proving consistency of the learned graph is beyond the scope of this paper.

Q2

Section 6 (Table 5) compares Transformer variants trained with the same Gaussian NLL. On the original Mixed Physics datasets, the full model achieves mean AUROC of 0.99. Replacing its shared adjacency with separate masks for each layer reduces AUROC to 0.77. Keeping the shared mask but replacing the log gate with an additive attention bias reduces it to 0.75. These comparisons isolate architectural changes while keeping the likelihood fixed.

Table 4 examines the complementary question by keeping the M2C backbone fixed and changing the forecasting objective. On independently sampled pure-variance graphs, AUROC rises from 0.532 with MSE to 1.000 with Gaussian NLL. We also compare against two models that capture conditional variance using their native objectives: RHINO matches M2C-NLL at 1.000 in this regime, and TS-LSNM scores 0.804.

Q3

We would not expect M2C to reliably recover an edge that changes only conditional skewness or other higher-order moments. This boundary follows directly from Appendix U.2. Lines 2273–2275 state: “Excluding XjX^j strictly increases optimal Gaussian NLL if and only if either moment changes on a set of positive probability; it strictly increases optimal MSE if and only if the conditional mean changes on such a set.” Appendix U.2 gives the difference in optimal expected Gaussian NLL between excluding and including source jj.

L−j⋆−Lfull⋆=E[DKL(N(μ,v)∥N(μ−j,v−j))]L^\star_{-j}-L^\star_{\mathrm{full}}=\mathbb{E}[D_{\mathrm{KL}}(\mathcal{N}(\mu,v)\Vert\mathcal{N}(\mu_{-j},v_{-j}))]

Here (μ,v)(\mu,v) and (μ−j,v−j)(\mu_{-j},v_{-j}) are the conditional mean and variance with and without the source. If the conditional mean and variance are unchanged by excluding jj, the two Gaussian predictors coincide and this difference is zero. The source can still carry directed information through changes in the shape of the conditional distribution, but Gaussian NLL provides no population signal for recovering that edge. We have also made the implication for higher-order-only dependencies explicit in the revised limitations (line 2432).

Thank you to the authors for their detailed responses and clarifications. I appreciate the additional architectural ablations and the acknowledgment of limitations concerning higher-order dependencies. However, my primary concern remains unresolved: the theoretical analysis establishes a population-level prediction result without providing guarantees for identifiability or consistency of the jointly learned causal graph, which is the central objective of the paper. Furthermore, RHINO achieves comparable performance on the pure-variance benchmark, leaving the distinctive advantage of M2C less clear. I also noticed an unresolved “??” placeholder in the caption of Table 4, which should be corrected. Overall, while the clarifications are helpful, they do not sufficiently address my concerns regarding the theoretical contribution and methodological novelty. Therefore, I would like to maintain my original score of 2.

We have addressed the other concerns in our general official comment above.

Thank you for pointing out the unresolved “??” in Table 4’s caption. This resulted from a \ref accidentally left during the rebuttal revision and has now been corrected in the updated manuscript.

Thank you to the authors for the further clarifications. However, my main concern remains the limited connection between the population-level theoretical analysis and the actual recovery of the learned causal graph. While I agree that a formal consistency guarantee is not necessarily required, the current theory offers limited insight into why the proposed joint optimization should recover the underlying causal structure. Although, now the empirical evidence is stronger, I remain unconvinced about the overall significance of the contribution.

Reviewer_yjvTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Mask2Cause learns a Granger-causality graph with a Transformer that represents each variable’s history as one token. A shared adjacency mask weights attention and is trained with a sparsity penalty. The model predicts conditional means and variances using Gaussian negative log-likelihood, aiming to detect dependencies that MSE can miss. The experiments cover several time-series benchmarks and introduce Mixed Physics, where mean-only and variance-only edges are separated.

Critical strengths
  1. The focus on variance dependencies makes sense. A source can change the conditional variance of a target without changing its conditional mean. Mixed Physics tests this directly and helps explain what the variance head adds.

  2. I like the variable-token design. Attention operates over variables, which helps with longer histories. Learning one shared mask also avoids having to combine separate attention heads into a graph after training.

Critical weaknesses
  1. My main concern is what the mask represents. With multiple layers, a source’s history can reach a target through another token even when their direct gate is zero (Appendix G). The log-mask floor also means a zero gate is not a hard block. The directed-information argument compares predictors with and without access to a source. It does not by itself establish that the jointly learned mask identifies Granger parents.

  2. The evaluation leaves several choices unclear. Appendix O does not state which validation metric selected hyperparameters and training duration. Tables 1–2 include self-loops that receive high scores by design, making cross-variable recovery harder to assess. On Mixed Physics, the five runs reuse one graph and trajectory per regime. Those error bars measure training variation but do not establish performance across graph and data realizations. Together, these issues make the reported gains difficult to judge.

Questions for authors
  1. Clarification: What validation metric selected hyperparameters and stopping epochs? Was ground-truth graph information used at any stage of selection?

  2. Clarification: Were the models retrained from scratch under each fixed mask in Appendix G.2? This matters because removing an edge after training does not test whether the network can learn to route that source’s information through another token.

  3. Evidence affecting my assessment: Can you report off-diagonal AUROC/AUPRC and repeat Mixed Physics across independently sampled graphs and trajectories, with tuning done on separate data? These results would help establish whether the gains hold beyond the current scoring choices and dataset realizations.

1 comment

W1

We do not claim that the directed-information result guarantees recovery of Granger parents. This is stated explicitly in Appendix M, line 1701: “We prove no consistency or identifiability result for the learned adjacency matrix.” Appendix U.2, lines 2307–2308, further clarifies: “This identifies the same population edges, not necessarily the same numerical loss reduction as DI, and does not assert recovery by a fitted network.” We use the theoretical result to motivate the loss and evaluate the learned mask empirically as a ranking of candidate parents. A low score suppresses the direct connection but does not rule out influence from the source. The log-mask floor leaves a small residual weight, and information can also pass through an intermediate token, as discussed in Appendix G. We have revised the manuscript (line 267) to make this absolutely clear.

Despite these routes, the learned mask ranks true edges well (as the experiments show). In Appendix G.2, the true mask also gives lower forecasting error than the mask with a direct edge removed in all six settings.

Q1

Yes. We used grid search to select the learning rate, sparsity penalty, and number of training epochs by validation AUROC. The ground-truth graph was used for model selection, but not in the training loss. For VAR, Lorenz-96, and RHINO’s Erdős–Rényi benchmark, we selected the settings on a calibration realization and kept them fixed across evaluation seeds. VAR and RHINO use new graphs at evaluation. Lorenz-96 uses new trajectories on the same graph. On DREAM3 and CausalTime, tuning and evaluation share the graph for each dataset.

Q2

Yes. For each setting in Table 13, we trained two models from scratch, one with the true mask and one with the ablated mask. The ablated model trained without the direct edge from the start, so it had the full training run to learn an indirect route. We have clarified this in Appendix G.2 (Updated manuscript, line 1161).

Q3

We included self-loops in Tables 1–2 to keep the scoring consistent with the published baseline results quoted there. We now report off-diagonal AUROC and AUPRC for both M2C variants in Appendix X in the revised manuscript. On Lorenz-96 and VAR, mean off-diagonal AUROC is at least 0.99 and AUPRC is at least 0.98 in every setting. Removing self-loops matters most on CausalTime Traffic: AUROC falls from 0.68 to 0.61 for M2C-MSE and from 0.71 to 0.65 for M2C-NLL. The corresponding off-diagonal AUPRCs are 0.50 and 0.53. Since we re-scored only M2C, we do not use these results to claim a ranking against the baselines in the off-diagonal setting.

We agree that using independently sampled graphs and trajectories is a better protocol. We repeated all three Mixed Physics regimes on 5 independently sampled graphs per regime, with a new trajectory for each graph. We kept the hyperparameters and training duration selected on the original data fixed. No new graph or trajectory was used for tuning. At variance-to-mean ratios of 50:50, 75:25, and 100:0, M2C-NLL achieved mean off-diagonal AUROCs of 0.996, 0.998, and 1.000, compared with 0.762, 0.653, and 0.532 for M2C-MSE. Table 4 in the revised manuscript reports means and standard deviations across the new graph–trajectory pairs.

Comments on the paper

3 comments

Thank you. The retraining clarification and the new mixed-physics runs partially address some of the concerns I had. Furthermore, your reply to my first question confirms that sparsity and training duration were selected using graph AUC, with tuning and evaluation sharing the same graph. Therefore, overall, I would need to see results with settings chosen without the evaluation graph, as well as baseline scores with self-loops excluded, before I would change my assessment; which, at this point, goes beyond what can be resolved during the rebuttal phase. Hence, I maintain my score.

We would like to clarify one point about the tuning protocol. Tuning and evaluation do not share the same graph on VAR, RHINO’s Erdős–Rényi benchmark, or the revised Mixed Physics evaluation. Thus, the concern about tuning and evaluation sharing a graph does not apply to these three benchmarks.

Our Lorenz-96 and DREAM3 evaluations use the fixed-topology systems and supplied networks also studied by many other papers including NGC and eSRU. Here we would like to mention that graph recovery metrics have also informed hyperparameter choices in prior work. For instance, NGC justifies its use of H=100 through experimental performance and reports the hidden-width comparison using Lorenz-96 AUROC in their Appendix A, Table 4.

Regarding self-loops, DREAM3 and Mixed Physics already exclude diagonal entries from our scoring. Tables 1–2 include them to retain comparability with the published baseline scores quoted from other papers.

AuthorsOpenReview ↗

Clarification of the paper’s contribution

We thank the reviewers for their engagement. We would like to make the scope of the contribution clear. Mask2Cause combines variable-token attention with a single adjacency matrix shared across layers and heads. The graph and forecasting model are learned jointly through conditional mean and variance prediction. Its contribution is this architecture, its empirical recovery and computational performance, and the Mixed Physics benchmark for distinguishing mean- and variance-driven dependencies.

The theoretical contribution is a population-level result explaining when Gaussian NLL detects dependencies that squared error misses. A consistency guarantee for the learned graph is beyond the scope of this paper. We believe the architectural and empirical contributions are substantive without such a guarantee. They are further supported by an extension that separates mean and variance parents, architectural ablations, robustness analyses, and downstream forecasting and root-cause localization experiments.

RHINO’s tie with M2C-NLL on pure-variance Mixed Physics concerns one of the seven settings in Table 4, where both achieve an off-diagonal AUROC of 1.000. This shows that both methods recover variance-driven dependencies, a capability we do not claim is unique to M2C. Across the full table, M2C-NLL matches or exceeds RHINO’s mean off-diagonal AUROC in six of seven settings, with measured training-run speedups of 3.5 - 95.8× under the selected configurations.

The revision further evaluates Mixed Physics on five independently sampled graph-trajectories per regime, with settings fixed using the original data. M2C-NLL achieves AUROCs of 0.996, 0.998, and 1.000, compared with 0.762, 0.653, and 0.532 for M2C-MSE. Mixed Physics, VAR, and RHINO ER therefore evaluate graphs that were not used for model selection. Whereas the other few benchmarks have intrinsic fixed-topologies that does not allow us to change the graph during tuning, as is the case for other baselines that used the benchmark.

We respectfully ask the reviewers and AC to consider the demonstrated architectural, empirical, and computational contributions, rather than treating a consistency guarantee or a unique ability to detect variance effects as claims of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.