acceptodds
Under review as a conference paper at ICLR 2027

Position: All Counterfactuals Are Wrong, and Few Are Useful

Abstract

_Counterfactuals_ — statements about what _might_ have happened under different circumstances — offer a natural way to phrase causal questions and have gained increasing prominence in contemporary machine learning with applications ranging from evaluating medical decisions to identifying discrimination. This position paper argues that, while counterfactuals are powerful philosophical constructs for describing and formalizing human causal reasoning, the computation of _individual_ counterfactual trajectories is riddled with dangers. In many practical scenarios, individualized counterfactuals are not truly useful or even harmful, either because the required assumptions cannot be adequately justified or because counterfactuals are conceptually misaligned with what would actually be informative for the task at hand.

Primary area:
causal reasoning
Keywords:
counterfactuals, structural causal models, causal inference, counterfactual identifiability, causal abstraction, model assumptions, counterfactual trajectories, time series, fairness, philosophy of science

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_U7cGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This position paper critically examines the use of individual counterfactuals, particularly how they are computed within Pearl's Structural Causal Models (SCMs). The central argument is that individual counterfactuals may be misleading or even harmful when the underlying models and assumptions do not support reliable counterfactual inference. The paper focuses primarily on individual counterfactual trajectories rather than group-level counterfactuals.

The critique is organized around three main claims. First, individual counterfactuals are often used in settings where alternative approaches, such as ex-ante prediction, decision-theoretic reasoning, expected utility, or group-level causal analysis, may be more appropriate. Second, computing individual counterfactuals requires strong assumptions about causal mechanisms, exogenous noise, cross-world consistency, and intervention semantics that are often difficult to justify. Third, individual counterfactuals are particularly sensitive to modeling choices, including structural assumptions, noise distributions, model simplifications, and parameter estimation errors. The authors argue that these limitations are generally less problematic for group-level causal analysis, where individual variations may average out.

The paper concludes with a set of practical recommendations encouraging researchers to carefully assess the suitability of counterfactual reasoning, explicitly justify their assumptions, and evaluate the sensitivity of counterfactual results to modeling choices.

Critical strengths

The paper addresses an important issue in causal inference and machine learning. As counterfactual reasoning is increasingly used in high-stakes applications, questioning the reliability and practical usefulness of individual counterfactuals is timely and relevant. It was among the main reasons I selected to review this paper.

The paper presents its position through three relatively well-structured claims covering the suitability of counterfactual reasoning, the assumptions required for its computation, and its sensitivity to modeling choices. This organization makes the main arguments easy to follow and provides a useful perspective on the limitations of individual counterfactual inference.

One of the more focused arguments concerns the interpretation and reuse of exogenous noise across factual and counterfactual worlds (Section 3.2). The authors highlight that noise variables may combine stable individual characteristics with realization-specific randomness, and that reusing the same noise under alternative interventions may not be conceptually justified. This is particularly relevant to neural SCMs and time-series counterfactual generation, where learned latent representations may not have a clear causal interpretation.

Critical weaknesses

My main concern is that the paper does not provide a sufficiently novel perspective or substantial contribution beyond existing critiques of counterfactual reasoning. The three main claims are largely well-known in the causal inference literature. The importance of selecting the appropriate causal estimand or reasoning framework, the reliance on strong and often untestable assumptions, and the sensitivity of counterfactual inference to model specification have all been extensively discussed. A position paper need not introduce a new algorithm or extensive experimental results, but it should advance a distinctive, well-supported position that meaningfully extends the existing discussion. I do not find this sufficiently established here.

Most of the paper consists of conceptual arguments and references to existing work, with relatively few concrete examples illustrating the practical implications of the claims. For instance, the predator-prey SCM introduced in Figure 1 provides an opportunity to demonstrate how plausible modeling choices can produce substantially different individual counterfactual trajectories, yet the paper does not develop this example into a quantitative analysis. Similarly, the treatment-response example in Figure 2 illustrates how averaging over heterogeneous individual effects can preserve population-level effects while distorting individual outcomes, but it remains a conceptual illustration. Even a small numerical experiment comparing different SCM specifications with similar observational or interventional behavior could make the argument substantially more convincing. The lack of such evidence makes it difficult to assess the practical significance of the identified limitations.

Several criticisms are not specific to individual counterfactuals or SCM-based reasoning. In particular, Claim B emphasizes the strong assumptions required for individual counterfactual inference. However, reliance on assumptions concerning causal structure, confounding, intervention semantics, and invariance is a fundamental characteristic of causal inference more generally. The paper does not consistently distinguish assumptions that are specific to individual-level counterfactual computation from those required for causal identification more broadly.

The practical recommendations are too general to be genuinely actionable. Section 4 recommends choosing the appropriate reasoning approach, explicitly stating assumptions, justifying the reuse of latent noise, and checking sensitivity to modeling choices and extrapolation. While these recommendations are reasonable, they largely reflect established good practices in causal modeling. They do not provide sufficiently concrete guidance on how researchers should implement them. For example, the paper could propose specific sensitivity analyses, criteria for determining when an individual counterfactual is insufficiently reliable, or a structured procedure for deciding between individual counterfactual inference, group-level causal estimation, and predictive approaches. Such guidance would strengthen the practical contribution of the paper.

Questions for authors

Could you use the same example to illustrate your claims in a more tangible way.

Reviewer_e7aNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This position paper argues that individual-level counterfactual trajectories are often unreliable or of limited practical utility. The authors identify three main issues: counterfactuals may address questions better answered by prospective predictions, rely on strong and often unjustified assumptions, and exhibit substantial sensitivity to modeling choices and uncertainty. Through conceptual examples and a synthesis of existing literature, the paper advocates more cautious use of individual counterfactuals and recommends alternative approaches, including population-level causal analysis, where appropriate.

Critical strengths
  • The paper points out that good observational predictive performance does not show that individual counterfactual trajectories are valid. Unlike a wrong forecast, a wrong counterfactual gives no error signal. This is especially relevant for neural causal models, where the learned latent variables may not have a clear causal interpretation.

  • The three claims give a useful way to check counterfactual claims. The recommendations in Section 4 ask researchers to justify their estimands, cross-world assumptions, and sensitivity to model specification. This can be useful for people developing or applying counterfactual ML methods.

Critical weaknesses
  1. Retrospective counterfactuals and prospective causal prediction are conflated. Section 2 explains the difference between ex-ante and counterfactual predictions, but the paper does not keep this difference when it discusses the cited work. Many of the cited time-series methods (e.g., Bica et al. 2020) estimate expected potential outcomes given the history at decision time. They do not infer noise from outcomes seen after the intervention. The paper should separate these two goals more clearly and explain how Claim A applies to these methods.

  2. The preference for group-level conclusions needs qualification. Averaging can reduce individual-level variation, but it does not solve confounding, identifiability, or model misspecification.

  3. Novelty and scope are not clear. Many of the main concerns (non-identifiability, untestable assumptions, sensitivity to model specification, extrapolation) are already well known, and the paper already cites much of these. The paper covers a wide range of work, but it is not clear what new idea or guidance it adds.

  4. The title and abstract are stronger than the main text. The paper says counterfactuals are "wrong" and possibly "harmful", but the introduction says they are not conceptually flawed and the conclusion accepts that they can be appropriate in some settings. There is no clear criterion for when a counterfactual is useful, and no systematic distinction between settings where it is defensible and settings where it is not.

Questions for authors
  1. Can the authors clearly separate retrospective SCM-based counterfactuals from prospective prediction of potential outcomes? Which of the cited neural treatment-effect models actually infer latent noise from post-intervention outcomes and keep it fixed across worlds?

  2. What criteria do the authors propose for deciding when an individual counterfactual is justified and useful? Can they give a concrete example where individual counterfactual reasoning is better than conditional interventional prediction, and explain how their criteria separate it from the problematic cases?

  3. Under what assumptions does aggregation reduce the problems in Claims B and C? Can the authors separate uncertainty that disappears with population averaging from non-identifiability that remains even with unlimited observational data?

Reviewer_WcJ4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that computing individual counterfactual trajectories in structural causal models often provides limited practical value and can produce misleading conclusions. It organizes the argument around three claims: counterfactuals may address questions better answered by prospective decision analysis; their interpretation depends on strong assumptions about noise, interventions, and cross-world relationships; and their outputs can be sensitive to modeling choices that are weakly constrained by observational or interventional fit. Particular attention is given to neural and temporal counterfactual models. The paper supports its position through conceptual examples and discussion of existing literature, and concludes with practical recommendations.

Critical strengths
  • The paper highlights an important gap between distributional fit and justified cross-world coupling. A model can fit observed distributions without establishing that its latent variables encode the background conditions that should remain fixed under an intervention. The discussion of stable individual attributes versus realization specific randomness makes this concern concrete and is particularly relevant to deep generative counterfactual models.

  • The paper connects assumptions to their intended use. Its distinction between assessing the quality of a decision using information available at decision time and retrospectively assessing causal responsibility is valuable. Encouraging researchers to specify the question before selecting a counterfactual construction could improve both interpretation and deployment in consequential applications.

Critical weaknesses
  • The argument establishes reasons for caution more convincingly than it establishes the claimed scarcity of practical utility. Non-identifiability, sensitivity, and untestable assumptions do not by themselves establish that an estimand is uninformative for a particular task. For example, a downstream conclusion may remain unchanged across multiple plausible cross-world couplings, even when individual trajectories differ. The paper gives limited attention to such cases, partial identification, or uncertainty-aware uses. Conversely, the suggestion that relevant modeling differences disappear through population averaging, especially in C.2, requires conditions: shared misspecification and intervention ambiguity need not cancel. A position paper need not introduce a new algorithm or experiment, but its central conclusion would be stronger with a worked comparison showing when counterfactual ambiguity changes a substantive conclusion and when the proposed alternative avoids that problem.
Questions for authors
  1. What criterion determines whether a counterfactual is “useful”? Would a decision or explanation that remains invariant across all models in a stated plausible class count as useful, despite non-identification of the individual trajectory? How would such cases affect the paper’s position?

  2. Under what assumptions do the modeling ambiguities discussed in C.2 average out at the population level? Please distinguish ambiguity that preserves all relevant interventional distributions from misspecification that can also change population-level causal effects.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.