acceptodds
Under review as a conference paper at ICLR 2027

When Observational Context Overrides Interventional Evidence in Transformers

Abstract

Can a transformer use interventional evidence when its context also contains conflicting observations? We study this question in synthetic structural causal models where observational association and the interventional effect have opposite signs, training decoder-only transformers from scratch on symbolic records while varying the training mixture and the context composition separately. We first formalize what each record type identifies: interventional probes determine the queried effect, observational records do not, and an ideal reader can still be harmed by observations on worlds where the two disagree. Against this reference, the composition of the context strongly shapes the answer. On 50 positive-effect evaluation worlds, a probe-only model produces 41 correct-sign slopes and 4 reversals, compared with 18 and 19 for a separately trained mixed-context model. At a single fixed checkpoint, adding observational records while keeping four probes reduces correct-sign slopes from 31 to 17 on 37 commonly parsed worlds and raises reversals from 1 to 11, an outcome that an ideal reader under the positive-effect training prior avoids, and replacing the observational values with zeros yields 25 correct-sign slopes of 33, compared with 16 for the world's own values. Activation patching shows that representations at the observational-record positions are sufficient to carry much of this effect in the tested subsets. The models also exhibit a positive-sign prior: on binary causal graphs, both models reverse on most negative-effect worlds, and sign-balanced retraining improves in-distribution sign reading for the probe-only model while leaving the out-of-distribution positive offset in place. Evaluations of causal learning should therefore vary inference-time evidence composition, not only the training mixture.

Primary area:
causal reasoning
Keywords:
causal direction, language models, interventional data, Simpson’s paradox, evidence-type competition, activation patching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XEwiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a transformer can use interventional evidence when the context also contains observational records with the opposite sign. The authors build synthetic causal models where the observational association is negative and the interventional effect is positive. Then they train 25.7M parameter transformers from scratch on symbolic record. This paper formalizes what each record type identifies and shows that an ideal reader under a positive-effect prior never reverses sign in Proposition 1. On 50 held-out worlds, the probe-only model has 41 hits and 4 reversals, compared with 18 and 19 for the mixed-context model. At a fixed checkpoint, increasing observational records from one to four reduces hits from 31/37 to 17/37 and raises reversals from 1 to 11.

Critical strengths
  1. The question is clear and important.

  2. The fixed-checkpoint design separates the effect of training from the effect of context at inference time (see Eq. 2, Table 4).

  3. Proposition 1 gives a simple test: under a sign-restricted prior, any reversal caused by added context indicates suboptimal evidence use.

Critical weaknesses
  1. The theoretical contribution is limited. Proposition 1 follows directly from the conditional expectation of a positive quantity, and Eq. (3) is a weighted average.

  2. The pooling model in Eq. (3) is not tested quantitatively. It gives a reversal threshold that depends on ω\omega, but ω\omega is never estimated. The closest check is in Appendix E, where the association between suppression and observational-correlation magnitude is not significant in the 50-world follow-up (p = 0.14). Table 4 reports only three dose levels, and record count changes with context length. The content experiment fixes the count but does not separate the two.

Questions for authors
  1. Could the authors fit Eq. (3) to the dose data and report the estimated ω\omega? This would show whether the pooling model fits the trained model quantitatively.

  2. Could the authors report more dose levels with context length held fixed, for example by padding shorter contexts? This would separate record count from context length.

Reviewer_TgRDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how observational evidence affects the ability of transformers to estimate causal effects from interventional data. The authors construct synthetic structural causal models in which observational associations and interventional effects have conflicting signs, and train decoder-only transformers using different combinations of observational and interventional records. A theoretical formulation is provided to distinguish the information available from each evidence type and to explain how observational context may interfere with interventional predictions. Experiments demonstrate that increasing observational records can reduce the accuracy of causal effect estimation and induce incorrect effect directions, even when informative interventional probes remain available. Additional analyses investigate the influence of observational content, training mixtures, activation patching, model scaling, and generalization to causal graphs with negative effects. The results suggest that transformers may rely excessively on observational associations when estimating causal effects, highlighting the importance of evaluating causal reasoning under conflicting evidence.

Critical strengths
  1. The paper addresses an interesting and relevant question concerning the integration of observational and interventional evidence in transformer-based causal inference. The distinction between observational association and causal effects is well motivated, and the controlled experimental setting allows the authors to investigate how conflicting evidence influences model predictions.

  2. The experimental design includes several useful analyses beyond a straightforward performance comparison. In particular, the fixed-checkpoint experiments help separate the effects of training configurations from inference-time context composition. The observational dose and content experiments, together with activation patching and sign-generalization analyses, provide empirical evidence that the observed failures are associated with how transformers process conflicting information.

Critical weaknesses
  1. The novelty and scientific contribution of the work are somewhat limited. The central observation that models may rely on misleading observational correlations despite the availability of more informative evidence is closely related to established findings on shortcut learning, spurious correlations, and conflicting evidence in in-context learning. Although the paper investigates this phenomenon in a causal inference setting, it remains unclear what fundamentally new understanding of transformer learning or causal reasoning is established beyond these existing observations.

  2. The theoretical analysis provides useful interpretations but offers limited explanatory depth. Proposition 1 follows directly from the positive-effect assumption and the linearity of conditional expectation, while the proposed pooling model is a simplified weighted combination of observational and interventional slopes. More importantly, the paper does not establish that the trained transformers actually implement this mechanism. The theoretical analysis therefore appears to explain the observed behavior qualitatively rather than reveal its underlying learning mechanism.

  3. The experimental evaluation is largely restricted to synthetic structural causal models and transformers trained from scratch using symbolic numerical representations. Although the authors include a larger model and a binary causal graph evaluation, these experiments do not establish whether the identified phenomenon generalizes to pretrained language models or more realistic causal reasoning tasks. The relevance of the conclusions to practical LLM-based causal inference therefore remains uncertain.

  4. The evaluation protocol and performance metrics raise additional concerns. The main evaluation uses 50 worlds with positive causal effects, where a simple positive-slope predictor can achieve perfect sign accuracy without using interventional evidence. Although the authors acknowledge this limitation and introduce a sign-generalization audit, both primary models perform poorly on negative-effect worlds. Furthermore, the fixed-checkpoint experiments exclude a non-negligible number of worlds due to parsing failures, and the larger-model results exhibit substantially attenuated predictions. These issues make it difficult to establish the extent to which the observed performance differences reflect genuine causal evidence integration rather than learned sign biases, output-format sensitivity, or training-related effects.

Questions for authors
  1. The paper argues that observational context can override interventional evidence in transformers. How does this finding differ fundamentally from existing observations concerning shortcut learning, spurious correlations, and conflicting demonstrations in in-context learning? The authors should clarify the specific new insight or mechanism contributed by this study.

  2. The theoretical pooling model provides an intuitive explanation of the experimental results, but it is unclear whether transformers actually integrate observational and interventional information according to this mechanism. Could the authors provide stronger empirical evidence connecting the theoretical model to learned transformer representations or attention behaviors?

  3. The main experiments focus on positive-effect worlds, and the authors acknowledge that a context-independent positive-slope predictor could achieve perfect sign accuracy. Could the authors conduct a more comprehensive evaluation using sign-balanced causal effects within the primary experimental distribution, rather than relying mainly on an out-of-distribution binary-SCM audit?

  4. The fixed-checkpoint experiments demonstrate that increasing observational records reduces correct-sign predictions. However, observational record count and context length increase simultaneously. Could the authors introduce length-matched controls, such as neutral or duplicated records, to better distinguish the influence of conflicting observational evidence from changes in context length and input structure?

  5. The evaluation mainly involves relatively small transformers trained from scratch on symbolic data. Could the authors investigate whether similar behavior occurs in pretrained transformers or language models using natural-language causal reasoning tasks? Such evidence would considerably strengthen the practical significance of the findings.

  6. The larger 0.93B-parameter models exhibit substantially smaller predicted slope magnitudes, and their performance cannot be directly compared with the smaller models because of differences in training and parsing behavior. Could the authors provide better-controlled scaling experiments to clarify whether increasing model capacity alleviates or exacerbates the identified phenomenon?

  7. The activation-patching experiments suggest that observational-record representations influence predictions, but the sparse component search does not identify a reliable circuit. Could the authors further investigate which components are responsible for the interference and whether the observed activation effects support a specific causal mechanism rather than general sensitivity to contextual information?

Reviewer_rdiTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how observational data might harm the causal inference results provided by transformers, when confounding variables are hidden and result in opposite correlation between treatment and outcome (Simpson Trap). This paper first proves that an “ideal reader” never return an estimate that reverse the effect sign under positive-effect prior. The paper then focus on the sign of causal effect (rather than magnitude), traning a GPT based model on purely interventional data, purely observational data (with specifically designed confounder causing Simpson Trap), and mixed data. The paper shows that more interventional data or strong confounding effect in interventional data will cause the trained GPT model more likely to learn reversed signs. The paper further shows that effect from observational records seem to mainly work on early and middle layers of the GPT class model. Also, interestingly, model trained on positive effects seem to learn a “bias towards positive sign” and such models will perform poorly in estimating negative ATE.

Critical strengths
  1. The writing of this paper is very clear. It formulate the classic Simpson traps using a SCM, and uses linear slope to illustrate the concept. Proposition 1 sets up the scope of the paper clearly: this paper is more focused on signs of causal effects rather than magnitude. Also, this paper is particular interested in “how the functional class of decoder-only transformer performs against the ideal-reader”. The definition of “hit rate” is straightforward and very clearly illustrated through Table 3 and 4

  2. Interestingly, this paper also shows that effect of misleading observational records in training are usually on early and middle layers. Furthermore, decoder-only transformers seem to suffer from the poor generalization of signs. Note that linear models will generalize well regardless of the effect sign in training is positive or negative, but decoder-only transformers seem to generalize poorly. Such insights are interesting.

Critical weaknesses
  1. Although the results from Section 5.1 and 5.3 are clear, they are not surprising. Given that the confounders are unobserved, when we construct observational data that leading to opposite correlation between X and Y, we should generally expect all causal models to suffer from low hit rate when there are more observational records, or confounders’ effect are stronger. We wouldn’t expect the transformer model to behave differently. Therefore, I wonder what is the innovation part of experiments in Section 5.1 and 5.3. Essentially the question is: we should not expect transformers to “smartly” differentiate interventional records and observational records in the first place, unless there is some mechanism ensuring that (prompt or redesign of GPT layers). Therefore, what is the innotvation part of showing Section 5.1 and 5.3 results? I would appreciate authors illustrate more on this (for instance, using the concept of “ideal reader”)

  2. This paper does not provide a clear illustration of what is the format of training data. Authors mention “vocabulary size 41” and “symbolic records”, but still I don’t know whether this data format is more similar to natural language or structural data records. Are the “symbolic records” essentially a more structural natural language (still not purely numerical records) We know that traditional causal inference or causal discovery follows this “data to model to result” pipeline, where the data is simply numerical records (rows and variables). However, if the input is natural language, then inevitably the pipeline becomes “data to language to model to result”. During the step of “language to model”, there is information loss or distortion. A natural question is: if the “symbolic records” are structural query fed to GPT model, then how generalized the results of this paper can be? If we re-write the training data in another way (with a different set of vocabulary), will all the results of this paper hold?

  3. Following point 2, we know that transformer model is sensitive to orders of data. If we randomly shuffle the orders in training data, will the results change? The paper does not seem to address this question.

Questions for authors

See weakness points 1 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.