- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This position paper critically examines the use of individual counterfactuals, particularly how they are computed within Pearl's Structural Causal Models (SCMs). The central argument is that individual counterfactuals may be misleading or even harmful when the underlying models and assumptions do not support reliable counterfactual inference. The paper focuses primarily on individual counterfactual trajectories rather than group-level counterfactuals.
The critique is organized around three main claims. First, individual counterfactuals are often used in settings where alternative approaches, such as ex-ante prediction, decision-theoretic reasoning, expected utility, or group-level causal analysis, may be more appropriate. Second, computing individual counterfactuals requires strong assumptions about causal mechanisms, exogenous noise, cross-world consistency, and intervention semantics that are often difficult to justify. Third, individual counterfactuals are particularly sensitive to modeling choices, including structural assumptions, noise distributions, model simplifications, and parameter estimation errors. The authors argue that these limitations are generally less problematic for group-level causal analysis, where individual variations may average out.
The paper concludes with a set of practical recommendations encouraging researchers to carefully assess the suitability of counterfactual reasoning, explicitly justify their assumptions, and evaluate the sensitivity of counterfactual results to modeling choices.
Critical strengths
The paper addresses an important issue in causal inference and machine learning. As counterfactual reasoning is increasingly used in high-stakes applications, questioning the reliability and practical usefulness of individual counterfactuals is timely and relevant. It was among the main reasons I selected to review this paper.
The paper presents its position through three relatively well-structured claims covering the suitability of counterfactual reasoning, the assumptions required for its computation, and its sensitivity to modeling choices. This organization makes the main arguments easy to follow and provides a useful perspective on the limitations of individual counterfactual inference.
One of the more focused arguments concerns the interpretation and reuse of exogenous noise across factual and counterfactual worlds (Section 3.2). The authors highlight that noise variables may combine stable individual characteristics with realization-specific randomness, and that reusing the same noise under alternative interventions may not be conceptually justified. This is particularly relevant to neural SCMs and time-series counterfactual generation, where learned latent representations may not have a clear causal interpretation.
Critical weaknesses
My main concern is that the paper does not provide a sufficiently novel perspective or substantial contribution beyond existing critiques of counterfactual reasoning. The three main claims are largely well-known in the causal inference literature. The importance of selecting the appropriate causal estimand or reasoning framework, the reliance on strong and often untestable assumptions, and the sensitivity of counterfactual inference to model specification have all been extensively discussed. A position paper need not introduce a new algorithm or extensive experimental results, but it should advance a distinctive, well-supported position that meaningfully extends the existing discussion. I do not find this sufficiently established here.
Most of the paper consists of conceptual arguments and references to existing work, with relatively few concrete examples illustrating the practical implications of the claims. For instance, the predator-prey SCM introduced in Figure 1 provides an opportunity to demonstrate how plausible modeling choices can produce substantially different individual counterfactual trajectories, yet the paper does not develop this example into a quantitative analysis. Similarly, the treatment-response example in Figure 2 illustrates how averaging over heterogeneous individual effects can preserve population-level effects while distorting individual outcomes, but it remains a conceptual illustration. Even a small numerical experiment comparing different SCM specifications with similar observational or interventional behavior could make the argument substantially more convincing. The lack of such evidence makes it difficult to assess the practical significance of the identified limitations.
Several criticisms are not specific to individual counterfactuals or SCM-based reasoning. In particular, Claim B emphasizes the strong assumptions required for individual counterfactual inference. However, reliance on assumptions concerning causal structure, confounding, intervention semantics, and invariance is a fundamental characteristic of causal inference more generally. The paper does not consistently distinguish assumptions that are specific to individual-level counterfactual computation from those required for causal identification more broadly.
The practical recommendations are too general to be genuinely actionable. Section 4 recommends choosing the appropriate reasoning approach, explicitly stating assumptions, justifying the reuse of latent noise, and checking sensitivity to modeling choices and extrapolation. While these recommendations are reasonable, they largely reflect established good practices in causal modeling. They do not provide sufficiently concrete guidance on how researchers should implement them. For example, the paper could propose specific sensitivity analyses, criteria for determining when an individual counterfactual is insufficiently reliable, or a structured procedure for deciding between individual counterfactual inference, group-level causal estimation, and predictive approaches. Such guidance would strengthen the practical contribution of the paper.
Questions for authors
Could you use the same example to illustrate your claims in a more tangible way.