acceptodds
Under review as a conference paper at ICLR 2027

Transformers are Limited Causal Learners

Abstract

Autoregressive Transformers trained by next-step prediction underlie foundation models across modalities and appear to learn the structure of the worlds that generate their data. A growing body of work reads this structure as causal, through attention maps or Transformer-based causal-discovery estimators, yet when prediction recovers causal structure, and where it fails, remains unclear. In this work, we examine this question for time series from lagged structural causal models, distinguishing a variable's predictive parents, on which its conditional distribution depends, from its structural parents, which enter its generating mechanism. We show that under four identifying conditions the two sets coincide, so that structural parents can be read from the gradient of the log-density that a Transformer learns by next-step prediction, without any graph objective. Beyond these conditions, where causes are hidden, effects are instantaneous, or regimes are pooled, other past variables become predictive parents, and reading them as direct causes yields causal illusions that no model capacity can remove. Experiments with decoder-only Transformers bear out both halves: within the identifying conditions, the readout recovers structural parents across mechanisms, noise, dimensions, and lag horizons, including parents that act only on the shape of the distribution, while attention reflects this structure only in shallow models. Outside them, recovery declines, most sharply under hidden regimes, and supplying the missing information through the regime identity or dedicated causal-discovery methods serves as a remedy. Taken together, our findings support viewing Transformers as causal learners within the identifying conditions and as limited causal learners beyond them, and they locate the limits of causal learning from prediction in the information that the observed history carries.

Primary area:
causal reasoning
Keywords:
Autoregressive Transformers, Causal Discovery, Causal Identifiability, Causal Structure Readout

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uxfhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper investigates whether, and under what conditions, the predictive parents identified by a transformer's gradient-based readout correspond to the structural causal parents of a target variable. The paper uses synthetic time-series data to evaluate the claim. The experiments show that the transformer can learn predictive dependencies corresponding to structural causal parents when the identification assumptions hold. The results on real-world benchmarks are provided in the appendix and provide inconclusive evidence for the main claim of the paper.

Critical strengths

The paper demonstrates that Transformers can achieve strong performance in identifying the causal parents of a target variable under certain assumptions. The proposed method outperforms several baselines on simulated datasets and achieves competitive performance on real-world-derived datasets. Although the Transformer architecture may not be the main factor contributing to these improvements, the paper provides an interesting investigation into the relationship between the predictive capabilities of ML models and their ability to identify causal parents from learned conditional distributions.

Critical weaknesses

Table 1 (in appendix) shows Transformer vs MLP comparison in causal discovery performance. It appears that the causal learning mechanism is not specific to transformers as MLP can use the same gradient readout to achieve better causal parent identification on AQI dataset. This may indicate transformer architecture is not the main factor for the performance improvement. Other strong conditional density estimators may achieve comparable performance using the same gradient readout.

In general, I feel the title and central framing of the paper may be misleading. Although the experiments demonstrate that transformer-based models can identify causal parents under certain assumptions, the theoretical result is not specific to transformers or the proposed gradient readout. Under the identifying assumptions, causal parents can be recovered through conditional dependence testing implemented using various nonlinear ML methods with or without gradient-based attribution.

Questions for authors

see weaknesses

5 comments

We thank the reviewer for recognizing our investigation linking learned predictive conditional distributions to structural causal parent identification. Below we directly address both critical weaknesses (W2 on title, central framing, and non-gradient conditional dependence testing; and W1 on Transformer vs. MLP and other conditional density estimators).

W2 (Part 1): Title and Central Framing ("The theoretical result is not specific to transformers or the proposed gradient readout")

We agree that Theorem 1 and Proposition 2 are architecture-agnostic, as they characterize the conditional distribution p∗(Xi,t∣Xt−L:t−1)p^*(X_{i,t} \mid X_{t-L:t-1}) targeted by next-step prediction. We center our framing and title on Autoregressive Transformers for two reasons:

  1. A unified perspective on when foundation models learn causality: Prior work on next-step Transformers is divided between studies reading their representations or attention maps as causal graphs [1, 2] and studies showing that accurate generative predictors can still misrepresent underlying mechanisms [3]. Our framework offers a unified view of when next-step Transformers recover structural parents (within A1–A4 via log-density gradients, reaching 0.998 AP on the same checkpoint where attention rollout drops to 0.321 AP) and when they fail (beyond A1–A4). Applying this readout to pretrained Chronos-2 [4] (native tokenization, context C=2048C = 2048, please see our response to Reviewer qpPt) further supports this link, reaching 0.764 zero-shot and 0.940 fine-tuned AP on Linear, and 0.814 fine-tuned AP on variance-only Scale.
  2. A simple, scalable causal discovery estimator as data and complexity grow: Fitting a Transformer by likelihood (DOT) scores all lagged candidates from one prediction fit without combinatorial search or sparsity tuning. While simpler models or CI tests can be stronger at 2K2\text{K}–9K9\text{K} samples, DOT overtakes MLP, GRU, and CNN on multi-mechanism Mixed and across scaled benchmarks from 25K25\text{K} to 120K120\text{K} steps (see W1, Table 1), and overtakes CMIknn by 5K5\text{K}–10K10\text{K} samples while running significantly faster (see W2 Part 2, Table 2).

Following ICLR policy, we cannot modify the title during the rebuttal period. If you still view the Transformer-centered framing as a main weakness after considering these additional experiment results, we will consider changing our title and update our framing to be architecture-agnostic (e.g., "Autoregressive Models Are Limited Causal Learners") in the final version.

W1: Transformer vs. MLP and Other Conditional Density Estimators Using the Same Gradient Readout ("MLP can use the same gradient readout to achieve better causal parent identification on AQI")

To compare conditional density estimators without sparsity penalties (λ=0\lambda = 0), we trained four architectures sharing the same mixture head, NLL objective, and gradient readout across synthetic (100K100\text{K}), released and expanded CausalTime (≈9K\approx 9\text{K}–100K100\text{K}, expanded via the same CausalTime data generation process), and physical Causal Chambers wind tunnel (120K120\text{K} real sensor steps [5]) benchmarks:

Table 1: Cross-Architecture Comparison Under Identical Mixture Head & Gradient Readout (All Pure NLL, λ=0\lambda = 0)

Benchmark (All Pure NLL, λ=0\lambda = 0)MetricDOT (Transformer)CNNGRUMLP
Synthetic Scale / Shape (100K100\text{K})AP1.000 / 1.0001.000 / 1.0001.000 / 1.0001.000 / 1.000
Synthetic Mixed (100K100\text{K})AP0.9980.9220.9060.855
CausalTime @ 9K9\text{K} (AQI / Traf / Med)AUROC0.689 / 0.643 / 0.6430.866 / 0.673 / 0.6260.819 / 0.663 / 0.5820.840 / 0.696 / 0.506
Expanded CausalTime @ 25K25\text{K} (AQI / Traf / Med)AUROC0.977 / 0.883 / 0.8130.974 / 0.775 / 0.6320.966 / 0.759 / 0.6150.911 / 0.751 / 0.555
Expanded CausalTime @ 50K50\text{K} (AQI / Traf / Med)AUROC0.991 / 0.909 / 0.8510.985 / 0.807 / 0.6560.971 / 0.790 / 0.6300.939 / 0.763 / 0.592
Expanded CausalTime @ 100K100\text{K} (AQI / Traf / Med)AUROC0.997 / 0.944 / 0.8710.993 / 0.852 / 0.7000.985 / 0.795 / 0.6560.953 / 0.795 / 0.623
Physical Wind Tunnel (120K120\text{K} real obs.)AP / AUROC0.526 / 0.7620.367 / 0.6080.357 / 0.6300.451 / 0.771

Note: On the Physical Wind Tunnel (chance AP=0.108\text{AP} = 0.108), classical baselines PCMCI, VAR-LiNGAM, and DYNOTEARS reach 0.229, 0.157, and 0.123 AP (0.748, 0.636, and 0.512 AUROC).

We see that at ≈9K\approx 9\text{K} steps, simpler models have lower estimation variance on AQI and Traffic, so CNN and MLP score higher there. From 25K25\text{K} steps onward, the ranking reverses on AQI and Traffic as well, and DOT leads all four architectures across all three CausalTime domains, multi-mechanism Mixed, and Wind Tunnel AP.


W2 (Part 2): Conditional Dependence Testing Without Gradient-Based Attribution (CMIknn and GPDC)

We also compared DOT against non-gradient nonlinear conditional independence tests (Tigramite PCMCI with CMIknn and GPDC) across all five nonlinear mechanisms at n∈{2K,5K,10K,100K}n \in \{2\text{K}, 5\text{K}, 10\text{K}, 100\text{K}\}. CMIknn and GPDC are more sample-efficient at n=2Kn = 2\text{K}, whereas DOT overtakes CMIknn on 3 of 5 mechanisms by n=5Kn = 5\text{K} and all 5 mechanisms by n=10Kn = 10\text{K} while running significantly faster (≈1\approx 1 min vs. 3.5–9.5 hours per run), avoiding the scalability bottleneck of nonparametric CI tests at larger nn:

Table 2: Nonparametric PCMCI (CMIknn, GPDC) vs. DOT Across Sample Sizes and Runtime

Mechanismn=2Kn = 2\text{K}: CMIknn / GPDC / DOTn=5Kn = 5\text{K}: CMIknn / DOTn=10Kn = 10\text{K}: CMIknn / DOTn=100Kn = 100\text{K}: DOT
Interaction0.347 / 0.314 / 0.1320.602 / 0.4730.710 / 0.7901.000
Scale0.704 / 0.954 / 0.3440.912 / 0.9720.966 / 1.0001.000
Shape0.603 / 0.560 / 0.1310.767 / 0.8960.863 / 0.9801.000
Transport0.331 / 0.574 / 0.1250.469 / 0.8020.652 / 0.9821.000
Mixed0.511 / 0.740 / 0.1160.732 / 0.7030.824 / 0.9090.998
Runtime / run12–45m / 3.3–4.5h / 0.2–0.3m1.2–3.4h / 0.5–0.8m3.5–9.5h / 0.8–1.3m8–10m

We hope these clarifications and experiments thoroughly address your concerns!

References

  • [1] Rohekar, Raanan Y., Yaniv Gurwicz, and Shami Nisimov. "Causal interpretation of self-attention in pre-trained transformers." Advances in Neural Information Processing Systems 36 (2023): 31450-31465.
  • [2] Nichani, Eshaan, Alex Damian, and Jason D. Lee. "How transformers learn causal structure with gradient descent." arXiv preprint arXiv:2402.14735 (2024).
  • [3] Vafa, Keyon, et al. "Evaluating the world model implicit in a generative model." Advances in Neural Information Processing Systems 37 (2024): 26941-26975.
  • [4] Ansari, Abdul Fatir, et al. "Chronos-2: From univariate to universal forecasting." arXiv preprint arXiv:2510.15821 (2025).
  • [5] Gamella, Juan L., Jonas Peters, and Peter Bühlmann. "Causal chambers as a real-world physical testbed for AI methodology." Nature Machine Intelligence 7.1 (2025): 107-118.

Dear Reviewer uxfh,

We want to kindly follow up regarding our rebuttal. We truly appreciate the time and effort you have already dedicated to reviewing our work and providing your feedback.

We have worked to address your concerns as follows:

Title and Central Framing (W2): We agree with you that our identifiability results are architecture-agnostic, since they characterize what any next-step predictor can recover from the observed history. We centered the paper on Transformers because whether next-step Transformers learn causal structure is still unsettled in the literature, and our results clarify when they do (the log-density gradient reaches 0.998 AP on the same checkpoint where attention rollout drops to 0.321 AP) and when they cannot. We also showed that the same readout extends to the pretrained Chronos-2 (a time-series foundation model). On reflection, since you and another reviewer independently regard the title as a misleading claim, we are happy to revise our framing in the revision and change to an architecture-agnostic title in the final version, since ICLR does not allow title changes during the discussion period. We are currently considering candidates such as "Autoregressive Models Are Limited Causal Learners" and "When Predictive Parents Are Structural Parents: Causal Structure from Next-Step Likelihood" (suggested by Reviewer qpPt).

Transformer vs. MLP and Other Density Estimators (W1): We trained Transformer, CNN, GRU, and MLP predictors with the same mixture head, the same pure likelihood objective, and the same gradient readout. We found that MLP and CNN lead on AQI and Traffic only at the released sample size, where simpler models have lower estimation variance. Once we expand CausalTime with its official data-generating process, the ranking reverses, and the Transformer becomes the best of the four architectures on all three domains at every sample size, as well as on the multi-mechanism Mixed system and in AP on the real wind tunnel data.

Conditional Dependence Testing Without Gradients (W2): We compared DOT against nonparametric PCMCI with CMIknn and GPDC on all five nonlinear mechanisms. These tests do recover the parents and are more sample-efficient at small sample sizes, whereas DOT overtakes CMIknn on all five mechanisms as the sample size grows, while taking about a minute instead of 3.5–9.5 hours per run.

You can find the full tables and details in our responses above. We would greatly appreciate it if you could let us know whether our responses have addressed your concerns. Thank you!

Best,

The Authors

Reviewer_oRKTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper investigates when autoregressive Transformers trained solely through next-step prediction can recover structural causal relationships in multivariate time series. The authors distinguish predictive parents from structural parents and establish four identifying conditions under which the two coincide. They propose extracting causal dependencies through the score energy of the learned conditional density, without explicit graph supervision. Experiments across synthetic systems show strong recovery under the identifying assumptions and demonstrate systematic failures under hidden confounders, instantaneous effects, and unobserved regimes. Additional analyses compare gradient-based attribution with attention-based readouts and evaluate the approach on causal discovery benchmarks.

Critical strengths
  1. Interesting theoretical characterization. The paper provides a principled connection between autoregressive prediction and structural causal discovery, explicitly characterizing the assumptions under which predictive dependencies can be interpreted as direct causal relationships.

  2. Comprehensive synthetic evaluation. The experiments cover different nonlinear mechanisms, noise distributions, variable dimensions, and lag horizons. Particularly interesting is the ability to recover dependencies that affect the conditional variance or distributional shape rather than the conditional mean. The experiments on hidden causes and regimes also effectively illustrate the theoretical limitations.

Critical weaknesses

The paper overall is interesting, however I think the main weakness is limited validation of the central claim in realistic settings. Most strong results are obtained from controlled synthetic systems satisfying the identifying assumptions. Although the paper includes CausalTime benchmarks, Appendix A.6.4 introduces gradient sparsity regularization and changes the model architecture, deviating from the claimed prediction-only setting. This makes it difficult to assess how reliably standard autoregressive Transformers recover causal structure in realistic scenarios without additional causal inductive biases, if I am understanding right.

Questions for authors

See weakness.

4 comments

We thank the reviewer for the positive assessment and for highlighting both our theoretical characterization and our synthetic evaluations across distributional mechanisms, hidden causes, and unobserved regimes.

W1 & Q1: Validation in Realistic Settings Without Gradient Sparsity Regularization or Architectural Changes

Our theoretical result supports that standard autoregressive Transformers trained by next-step likelihood alone (λ=0\lambda = 0), with no graph objective or sparsity penalty, recover causal structure once the identifying conditions are fulfilled and data support accurate density estimation. In small-sample scenarios such as the released CausalTime benchmark (≈9K\approx 9\text{K} steps across 2020–3636 variables), using a 1-layer backbone and sparsity regularization in our submission was a standard way to reduce small-sample parameter uncertainty. We appreciate the insightful suggestions and provide more realistic setting experiments below, with our standard 4-layer Transformer (DOT). Once sample size and model capacity scale up, pure prediction-only (λ=0\lambda = 0) is sufficient on its own, and adding sparsity regularization instead introduces a slight degradation. Below we demonstrate this on CausalTime from ≈9K\approx 9\text{K} to 100K100\text{K} steps (1) and on a 120K120\text{K} real physical system (2).

1. CausalTime Across Sample Sizes (≈9K→25K→50K→100K\approx 9\text{K} \to 25\text{K} \to 50\text{K} \to 100\text{K} Steps, AUROC)

Following the officially released CausalTime data-generating process, we expanded the dataset from the released ≈9K\approx 9\text{K} steps to 25K25\text{K}, 50K50\text{K}, and 100K100\text{K} steps and evaluated all models across 5 seeds. We see that from 25K25\text{K} steps onward, unmodified DOT (λ=0\lambda = 0) achieves the highest AUROC and AUPRC among all methods across all three domains at every sample size, whereas adding sparsity regularization degrades recovery accuracy:

Predictor on CausalTimeSparsityAQI (≈9K→25K→50K→100K\approx 9\text{K} \to 25\text{K} \to 50\text{K} \to 100\text{K})Traffic (≈9K→25K→50K→100K\approx 9\text{K} \to 25\text{K} \to 50\text{K} \to 100\text{K})Medical (≈9K→25K→50K→100K\approx 9\text{K} \to 25\text{K} \to 50\text{K} \to 100\text{K})
DOT (Transformer)None (λ=0\lambda = 0)0.689 →\to 0.977 →\to 0.991 →\to 0.9970.643 →\to 0.883 →\to 0.909 →\to 0.9440.643 →\to 0.813 →\to 0.851 →\to 0.871
DOT (Transformer) + SparsityPenalized0.796 →\to 0.921 →\to 0.874 →\to 0.9410.740 →\to 0.869 →\to 0.894 →\to 0.9170.637 →\to 0.799 →\to 0.844 →\to 0.847
CNN (same MoG head)None (λ=0\lambda = 0)0.866 →\to 0.974 →\to 0.985 →\to 0.9930.673 →\to 0.775 →\to 0.807 →\to 0.8520.626 →\to 0.632 →\to 0.656 →\to 0.700
GRU (same MoG head)None (λ=0\lambda = 0)0.819 →\to 0.966 →\to 0.971 →\to 0.9850.663 →\to 0.759 →\to 0.790 →\to 0.7950.582 →\to 0.615 →\to 0.630 →\to 0.656
MLP (same MoG head)None (λ=0\lambda = 0)0.840 →\to 0.911 →\to 0.939 →\to 0.9530.696 →\to 0.751 →\to 0.763 →\to 0.7950.506 →\to 0.555 →\to 0.592 →\to 0.623

2. Real Physical System (Causal Chambers Wind Tunnel [1], 120K120\text{K} Steps)

We evaluated unmodified DOT (λ=0\lambda = 0) on physical recordings from the Causal Chambers wind tunnel [1] (5 random-walk actuators driving 11 physical sensors every 0.3 s0.3\text{ s}; 120K120\text{K} training steps, 2626 true edges among 240240 pairs, chance AP=0.108\text{AP} = 0.108). Standard DOT achieves the highest AP across all neural and classical baselines and matches comparably to the highest AUROC:

Method on Physical Wind Tunnel (120K120\text{K} obs.)Sparsity (λ\lambda)APAUROC
DOT (Transformer)None (λ=0\lambda = 0)0.526 ±\pm 0.0360.762 ±\pm 0.041
MLP (same MoG head & readout)None (λ=0\lambda = 0)0.451 ±\pm 0.0330.771 ±\pm 0.030
CNN (same MoG head & readout)None (λ=0\lambda = 0)0.367 ±\pm 0.0060.608 ±\pm 0.015
GRU (same MoG head & readout)None (λ=0\lambda = 0)0.357 ±\pm 0.0170.630 ±\pm 0.025
PCMCI (ParCorr)—0.2290.748
VAR-LiNGAM—0.1570.636
DYNOTEARS—0.1230.512

We hope these pure prediction-only and standard DOT results thoroughly address your concern!

Reference

  • [1] Gamella, Juan L., Jonas Peters, and Peter Bühlmann. "Causal chambers as a real-world physical testbed for AI methodology." Nature Machine Intelligence 7.1 (2025): 107-118.

Dear Reviewer oRKT,

We want to kindly follow up regarding our rebuttal. We truly appreciate the time and effort you have already dedicated to reviewing our work, and we are grateful for your positive assessment.

To address your main concern about validation in realistic settings without sparsity regularization or architectural changes:

  • We clarified that the 1-layer backbone and the sparsity penalty in Appendix A.6.4 were only used to reduce small-sample uncertainty on the released CausalTime benchmark, and that our claim does not depend on them.
  • We expanded CausalTime with its official data-generating process and evaluated our standard 4-layer Transformer trained by next-step likelihood alone. From the first expanded sample size onward, this unmodified model achieves the highest AUROC among all methods on all three domains, whereas adding the sparsity penalty degrades the recovery.
  • We also evaluated the same prediction-only model on real recordings from the Causal Chambers wind tunnel, where it achieves the highest AP among all neural and classical baselines.

You can find the full tables and details in our response above. We would greatly appreciate it if you could let us know whether our response has addressed your concern. Thank you!

Best,

The Authors

Thank you for the additional experiments. The new results on larger-scale CausalTime and the real-world wind tunnel data have addressed my concerns about empirical validation, so I have raised my soundness and informativeness score accordingly.

AuthorsOpenReview ↗

Thank you for the insightful feedback and for raising the scores

Thank you for the constructive feedback and for raising the soundness and informativeness scores. Your question about the prediction-only setting on realistic data led to the expanded CausalTime study and the Causal Chambers wind tunnel experiment, which strengthen the paper's empirical validation. We will include these results in the revision.

Reviewer_qpPtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper asks when the past variables a forecaster relies on are the same as the variables that actually generate the target. It proves they coincide under four standard conditions (no hidden common causes or same-instant effects, a long enough lookback, and no exact cancellations). It also shows that when any condition fails, other past variables that carry information about the missing piece get pulled in as spurious causes in a way that no amount of data or model size can remove. Empirically, a small transformer trained only to predict the next step on simulated systems recovers the true causes almost perfectly when the conditions hold, including causes that change only the spread or shape of the target rather than its average, which standard methods miss entirely. When the conditions are broken, recovery falls in proportion to how badly (down to 0.32 with ten hidden causes, 0.48 with nine same-instant effects, 0.17 to 0.57 with ten pooled regimes) even though the model’s forecasts remain essentially perfect. Telling the model which regime it is in, or passing its candidate list to a classical discovery method, restores most of the loss. Attention weights reflect the structure only in a one-layer model. A real-data check on air-quality, traffic, and medical benchmarks uses a different, smaller model with an added sparsity penalty.

Critical strengths
  1. One of the paper's strengths is that it doesn't just claim that a prediction model exposes the true causes. The authors check them edge by edge. For three systems, where the correct answer can be computed, the paper compares the model’s sensitivity to each possible cause against the ground truth. That turns a theoretical statement into an empirically verified statement, which is somewhat rare in the literature.
  2. The paper also shows something that standard methods do not: some causes change only how spread out or how skewed the target is, not its average. The usual discovery methods look at averages and miss those causes completely. There is also a flip side: when the data hide a cause or pool different regimes, the model keeps forecasting almost perfectly while its picture of the wiring goes wrong. This is exactly the paper's point: good prediction does not imply correct causal understanding.
Critical weaknesses
  1. My biggest issue is that the paper makes a comparison against methods that cannot detect spread-only or shape-only causes by design. Beating such methods does not say much about the method's usefulness. At the same time, there are citations in the paper to the nonparametric independence tests and existing gradient-based neural methods, which would provide the appropriate grounds for comparison, but the tests are not run. So while I find the paper's design elegant in principle, I can't trust it because I don't have anything relevant to compare it to.
  2. The title is not appropriate for what the paper delivers. See my note in the "Questions for Authors" section.
Questions for authors

I want to say upfront that I am an expert in TSFM, not transformers, so the paper is not very well matched to me. Therefore, my review is not very confident, and the scores can be adjusted if the questions I ask are amply answered.

  1. My first question is about the paper title, and it comes from my potential misunderstanding of the paper's contribution. While the "Transformers are Limited Causal Learners" is accurate, it has a misleading emphasis. To me, the fact that transformers are causal learners with certain limitations sounds the same as "Toyotas can drive on highways." A better title should reflect what the paper is really about: the limit the paper identifies belongs to the observed data, not to the Transformer that happens to be used. So a better title would name prediction or predictive parents as the subject. For example, “When Predictive Parents Are Structural Parents: Causal Structure from Next-Step Likelihood.” Because I don't work on transformers, I may be missing minute details, but you get the idea. If the rules allow, would the authors be open to changing the title?

  2. On comparison with appropriate methods. On Scale, Shape, Transport, and Mixed at 100K samples, what AP do PCMCI with a nonparametric test (CMIknn or GPDC in Tigramite) and a Jacobian-based neural method (JRNGC or Neural Granger with the same mixture head) achieve? The current references cannot detect these parents by construction, so the reported gap does not establish the study's advantage.

  3. Can the gradient-based attribution be applied to a pretrained forecaster (Chronos-2 or TimesFM, with its native tokenization) on any of the synthetic systems? A positive result would make the architecture-specific framing accurate; a negative one would still be informative and would justify changing the title as I suggested.

8 comments
AuthorsOpenReview ↗

Official Response to Reviewer qpPt

We thank the reviewer for the thoughtful feedback and for highlighting our edge-by-edge score-energy verification as well as the distinction between accurate forecasting and causal understanding. Below we address each weakness and question directly (W1 & Q2, Q3, and W2 & Q1).

W1 & Q2: Comparison with Appropriate Methods (Nonparametric PCMCI and Jacobian / Neural Granger Methods with the Same Mixture Head)

We evaluated DOT against both suggested baseline families across all five nonlinear mechanisms (Interaction, Scale, Shape, Transport, and Mixed). At n=100Kn = 100\text{K}, DOT matches or outperforms the Neural Granger and Jacobian baselines both in their native forms and with the same mixture head (A); against nonparametric PCMCI, CI tests lead at n=2Kn = 2\text{K}, while DOT overtakes CMIknn by n=5Kn = 5\text{K}–10K10\text{K} at much lower runtime (B).

A. Jacobian (JRNGC-F) and Neural Granger (cMLP): Native vs. Same Mixture (MoG) Head (AP)

Mechanism (n=100Kn = 100\text{K})DOTcMLPcMLP (MoG head)JRNGC-FJRNGC-F (MoG head)PCMCI (ParCorr)
Interaction1.000 ±\pm 0.0000.171 ±\pm 0.0440.068 ±\pm 0.0130.072 ±\pm 0.0130.088 ±\pm 0.0250.069 ±\pm 0.015
Scale1.000 ±\pm 0.0000.072 ±\pm 0.0211.000 ±\pm 0.0000.662 ±\pm 0.0650.989 ±\pm 0.0170.091 ±\pm 0.008
Shape1.000 ±\pm 0.0000.074 ±\pm 0.0231.000 ±\pm 0.0000.071 ±\pm 0.0070.957 ±\pm 0.0190.074 ±\pm 0.019
Transport1.000 ±\pm 0.0000.073 ±\pm 0.0211.000 ±\pm 0.0000.066 ±\pm 0.0020.944 ±\pm 0.0250.102 ±\pm 0.054
Mixed0.998 ±\pm 0.0020.077 ±\pm 0.0220.870 ±\pm 0.0650.283 ±\pm 0.0320.751 ±\pm 0.0840.176 ±\pm 0.024

B. Nonparametric PCMCI (CMIknn and GPDC in Tigramite) Across Sample Sizes and Runtime

Because the complexity of CMIknn and GPDC scales superlinearly with nn, we only evaluated them alongside DOT on identical sample prefixes of n∈{2K,5K,10K}n \in \{2\text{K}, 5\text{K}, 10\text{K}\}, plus n=100Kn = 100\text{K}. They are more sample-efficient at small sample sizes (n=2Kn = 2\text{K}), whereas DOT overtakes CMIknn by n=5Kn = 5\text{K}–10K10\text{K} without their scalability bottleneck at larger scales (especially GPDC):

Mechanismn=2Kn = 2\text{K}: CMIknn / GPDC / DOTn=5Kn = 5\text{K}: CMIknn / DOTn=10Kn = 10\text{K}: CMIknn / DOTn=100Kn = 100\text{K}: DOT
Interaction0.347 / 0.314 / 0.1320.602 / 0.4730.710 / 0.7901.000
Scale0.704 / 0.954 / 0.3440.912 / 0.9720.966 / 1.0001.000
Shape0.603 / 0.560 / 0.1310.767 / 0.8960.863 / 0.9801.000
Transport0.331 / 0.574 / 0.1250.469 / 0.8020.652 / 0.9821.000
Mixed0.511 / 0.740 / 0.1160.732 / 0.7030.824 / 0.9090.998
Runtime / run12–45m / 3.3–4.5h / 0.2–0.3m1.2–3.4h / 0.5–0.8m3.5–9.5h / 0.8–1.3m8–10m

Q3: Applying Gradient-Based Attribution to a Pretrained Forecaster (Chronos-2 with Native Tokenization)

We applied gradient-based attribution to Chronos-2 [1] on three settings (Linear, Scale, Mixed) with its native tokenization across context lengths C∈{32,128,512,2048}C \in \{32, 128, 512, 2048\}, both zero-shot and after official fine-tuning at C=2048C = 2048:

System (Chance AP)ModeC=32C = 32 (AP / AUROC)C=128C = 128 (AP / AUROC)C=512C = 512 (AP / AUROC)C=2048C = 2048 (AP / AUROC)
Linear (0.055)Zero-shot0.157 / 0.5620.208 / 0.6930.389 / 0.8560.764 / 0.972
Fine-tuned0.179 / 0.6800.401 / 0.8480.850 / 0.9500.940 / 0.988
Scale (0.062)Zero-shot0.076 / 0.5150.074 / 0.5280.090 / 0.5830.122 / 0.707
(variance-only)Fine-tuned0.125 / 0.6710.322 / 0.7970.623 / 0.9250.814 / 0.968
Mixed (0.062)Zero-shot0.083 / 0.5150.089 / 0.5500.101 / 0.5380.099 / 0.592
Fine-tuned0.121 / 0.5270.146 / 0.5950.169 / 0.6070.186 / 0.617

Chronos-2 recovers causal parents zero-shot on Linear (0.764 AP / 0.972 AUROC) and reaches 0.940 AP / 0.988 AUROC after fine-tuning, alongside 0.814 AP / 0.968 AUROC on variance-only Scale. Longer contexts (C=32→2048C = 32 \to 2048) improve both forecasts and structure recovery (lifting zero-shot AUROC to 0.972 on Linear and 0.707 on Scale, and fine-tuned AP from 0.179 to 0.940 and 0.125 to 0.814). On Mixed, Chronos-2's pinball loss captures fine-grained shape changes in complex Mixed data generation process less sharply than a mixture density head, which supports our theoretical result: the gradient readout can recover only the distributional dependencies that the predictive model represents.

W2 & Q1: Paper Title ("Transformers are Limited Causal Learners") and Central Framing

We appreciate your "Toyotas can drive on highways" analogy, and we agree that the population identifiability limit in Section 3 belongs to the observed history. We centered our title and framing on Autoregressive Transformers for two reasons:

  1. A unified perspective on when foundation models learn causality: Unlike cars built for highways by design, whether next-step Transformers learn causal structure is unsettled: some works read their representations or attention maps as causal graphs [2, 3], while others show that accurate generative predictors can still misrepresent underlying mechanisms [4]. Our work clarifies when next-step Transformers recover structural parents (within A1–A4 via log-density gradients, at 0.998 AP vs. 0.321 AP for attention on the same checkpoint), when observational limits create causal illusions (beyond A1–A4), and, in Q3 above, that this framework extends to pretrained Chronos-2 [1].
  2. A simple, scalable causal discovery estimator as data and complexity grow: Fitting one Transformer by likelihood scores all lagged candidates without combinatorial search (significantly faster than CMIknn and GPDC in W1/Q2B). Comparing DOT against CNN, GRU, and MLP under the exact same mixture head, pure likelihood objective (λ=0\lambda = 0), and gradient readout, simpler models can lead on AQI and Traffic at ≈9K\approx 9\text{K} samples, whereas DOT leads on multi-mechanism Mixed, across all three expanded CausalTime domains from 25K25\text{K} to 100K100\text{K} steps, and in AP on the 120K120\text{K} Causal Chambers wind tunnel [5]:
Benchmark (All Pure NLL, λ=0\lambda = 0)MetricDOT (Transformer)CNNGRUMLP
Synthetic Scale / Shape (100K100\text{K})AP1.000 / 1.0001.000 / 1.0001.000 / 1.0001.000 / 1.000
Synthetic Mixed (100K100\text{K})AP0.9980.9220.9060.855
CausalTime @ 9K9\text{K} (AQI / Traf / Med)AUROC0.689 / 0.643 / 0.6430.866 / 0.673 / 0.6260.819 / 0.663 / 0.5820.840 / 0.696 / 0.506
Expanded CausalTime @ 25K25\text{K} (AQI / Traf / Med)AUROC0.977 / 0.883 / 0.8130.974 / 0.775 / 0.6320.966 / 0.759 / 0.6150.911 / 0.751 / 0.555
Expanded CausalTime @ 50K50\text{K} (AQI / Traf / Med)AUROC0.991 / 0.909 / 0.8510.985 / 0.807 / 0.6560.971 / 0.790 / 0.6300.939 / 0.763 / 0.592
Expanded CausalTime @ 100K100\text{K} (AQI / Traf / Med)AUROC0.997 / 0.944 / 0.8710.993 / 0.852 / 0.7000.985 / 0.795 / 0.6560.953 / 0.795 / 0.623
Physical Wind Tunnel (120K120\text{K} real obs.)AP / AUROC0.526 / 0.7620.367 / 0.6080.357 / 0.6300.451 / 0.771

Note: On the Physical Wind Tunnel (chance AP=0.108\text{AP} = 0.108), classical baselines PCMCI, VAR-LiNGAM, and DYNOTEARS reach 0.229, 0.157, and 0.123 AP (0.748, 0.636, and 0.512 AUROC). CausalTime is expanded using the same officially released data-generating process.

Following ICLR policy, we cannot modify the title during the rebuttal period. If policy permits, we will consider changing our title to be architecture-agnostic (e.g., "Autoregressive Models Are Limited Causal Learners" or "When Predictive Parents Are Structural Parents: Causal Structure from Next-Step Likelihood") in the final version if you still view the Transformer-specific title as a main weakness after reviewing these additional experiment results.

We hope these new experiments and clarifications amply address your questions!

References

  • [1] Ansari, Abdul Fatir, et al. "Chronos-2: From univariate to universal forecasting." arXiv preprint arXiv:2510.15821 (2025).
  • [2] Rohekar, Raanan Y., Yaniv Gurwicz, and Shami Nisimov. "Causal interpretation of self-attention in pre-trained transformers." Advances in Neural Information Processing Systems 36 (2023): 31450-31465.
  • [3] Nichani, Eshaan, Alex Damian, and Jason D. Lee. "How transformers learn causal structure with gradient descent." arXiv preprint arXiv:2402.14735 (2024).
  • [4] Vafa, Keyon, et al. "Evaluating the world model implicit in a generative model." Advances in Neural Information Processing Systems 37 (2024): 26941-26975.
  • [5] Gamella, Juan L., Jonas Peters, and Peter Bühlmann. "Causal chambers as a real-world physical testbed for AI methodology." Nature Machine Intelligence 7.1 (2025): 107-118.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Review Summary

We sincerely thank all reviewers for their constructive feedback. We are encouraged that Reviewer oRKT recognized the principled connection between autoregressive prediction and structural causal discovery, that Reviewer uxfh found the paper an interesting investigation into the relationship between predictive capability and the identification of causal parents, that Reviewer qpPt highlighted the edge-by-edge verification of the learned score energy against the ground truth, and that Reviewers qpPt and oRKT valued the comprehensive synthetic evaluation, including the recovery of parents that act only on the variance or shape of the distribution. We are grateful that all three reviewers responded positively to the new results and raised their scores.

The main revisions are as follows, highlighted in blue in the revised manuscript:

  • Appropriate baselines. PCMCI with nonparametric tests (CMIknn, GPDC) and neural Granger methods (cMLP, JRNGC-F), also with our mixture head: Section 4.1, Table 1, Appendices A.6.7 and A.6.8.
  • Architectures. Transformer against CNN, GRU, and MLP under the same likelihood and readout (Figures 4 and 5, Appendix A.6.9); the findings hold when all predictors have the same number of parameters (Tables 10, 14, and 16).
  • Realistic data without sparsity. The standard four-layer Transformer (DOT) on expanded CausalTime and on the Causal Chambers wind tunnel: new Section 4.4, Appendices A.6.11 and A.6.12.
  • Pretrained forecaster. The readout applied to Chronos-2, zero-shot and fine-tuned: Section 4.4, Appendix A.6.10.
  • Training objective comparison. Squared-error, Gaussian, and mixture heads: Table 1.
  • Presentation. The four findings in the introduction now follow the four subsections of Section 4, the real-world results moved from the appendix into the main text, and the appendix has one subsection per experiment with a table of the values behind each main-text figure.

To Reviewer qpPt. We added the comparisons you asked for. Nonparametric PCMCI leads at 2K samples, and from 10K samples DOT leads CMIknn on all five mechanisms at a fraction of the run time (Section 4.1, Appendix A.6.7). Neural Granger methods with the same mixture head are comparable to DOT on Scale, Shape, and Transport and stay behind on Interaction and Mixed (Section 4.3, Appendix A.6.8). We also applied the readout to pretrained Chronos-2 (Section 4.4, Appendix A.6.10), documented the baseline search protocol (Appendix A.2.2), and added a discussion of sparsity penalties, as you suggested (Appendix A.6.11).

To Reviewer oRKT. We evaluated the unmodified four-layer Transformer, trained by likelihood alone, on CausalTime expanded to 25K–100K steps and on real wind-tunnel recordings (Section 4.4, Appendices A.6.11 and A.6.12). Without a sparsity penalty, it has the highest AUROC of all predictors on every CausalTime system at these sizes and the highest AP on the wind tunnel.

To Reviewer uxfh. We agree that the identifiability result is architecture-agnostic. Under the same mixture head, likelihood, and readout, simpler predictors lead on AQI and Traffic at 9K steps, and the Transformer has the highest AUROC from 25K steps onward on all three CausalTime systems and leads on Mixed at 100K samples, in AP on the wind tunnel, and under hidden regimes at 1M samples (Sections 4.3 and 4.4, Appendices A.6.9, A.6.11, and A.6.12). We also added the comparison with conditional-independence tests that use no gradients (Section 4.1, Appendix A.6.7).

Title (Reviewers qpPt and uxfh). After our discussion, neither reviewer raised further concerns about the title or the framing. We therefore keep the Transformer-centered framing, now supported by the scaling and Chronos-2 results, and remain open to an architecture-agnostic title in the final version.

Please review our detailed responses to each point raised. Thank you once again for your time and insightful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.