- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces two new definitions of counterfactuals, with the intention of leveraging the notions for providing counterfactual explanations, for instance in settings such as algorithmic recourse (when flipping of a decision is concerned).
Critical strengths
(S1) Some of the thinking in the paper is interesting. The paper explores a possibly worthwhile formulation for counterfactual explanations.
Critical weaknesses
(W1) The motivating example is difficult to follow, and does not seem to provide crystal-clear motivation as one would hope.
The example describes the setting with X -> A -> Y, and asks the question about alternative values of , labeled , which may result in a positive decision. Then the example states this is considered through a revision law which conditions on : however, it is not clear why there is no conditioning on in this law? I could understand the query motivation much better with behind the conditioning bar, which actually happens later on in the paper (Section 5, where conditioning on history appears).
Similarly, an alternative is discussed, where appears suddenly, once again without a crisp reason.
Overall, this leaves the feeling that the introductory example could be much better.
(W2) Why is the restriction to a good idea?
The u-FAC definition seems to be a Layer 3 analogue of the concept of soft-intervention in Layer 2, which instead of atomic interventions, allows for distributions over the intervened value.
However, u-FAC conceptually departs heavily from this. Firstly, why does the definition require to condition on ? For instance, a soft-intervention may allow dependence on a new parent, whereas u-FAC does not; also, in Layer 3, one may consider distributions which depend on other downstream variables (e.g., in the motivating example), yielding another restriction which is not clearly motivated.
Furthermore, the with proposal also seems to be a pretty arbitrary choice, which also depends on a specific notion of discrepancy.
Additionally, soft interventions also lead to the -calculus, which may help solve questions of identification, whereas this does not seem to be the case for this paper, beyond a non-ID counterexample in Thm. 1 and ID-by-construction in Prop. 5.
Therefore, this key definition seems to be tailored w.r.t. a downstream problem that is being solved (to satisfy something like a Bayesian factorization property), rather than being motivated as a fundamental new notion of a counterfactual. Put differently, the paper’s ideas would probably be leveraged much better if it was framed as a method for solving a specific explanation/recourse problem, rather than introducing a new notion of a counterfactual.
(W3) **Is Def. 2 even a counterfactual quantity? **
From reading Definition 2, it seems unclear whether this construct is actually a Layer 2, interventional construct, rather than a counterfactual. If yes, the paper stretches the name of “counterfactual” too far, and does not really sit well with existing literature.
Relatedly, for the case where the “null revision” is used (as in Prop. 1), doesn’t r-FAC violate the consistency axiom for SCMs, since can happen after resampling? It would be important to discuss the implication of this possible side effect.
(W4) Does the discrepancy form allow multivariate notion of distance/cost in the recourse formulation?
In the formulation, discrepancy of to is considered. However, a cost of changing one variable may depend on its children (e.g., V1 = number of credit lines -> V2 = total debt on credit lines, where the cost of changing V1 may depend on V2). Can the formulation be used to model such scenarios?
Questions for authors
(Q1) Can the authors point to the exact results that require the most significant theoretical lifting?
4 comments
Rebuttal to Reviewer ArHH: 2/2
What Definition 2 represents under a null revision (W3)
Is r-FAC a Layer 2 intervention? In general, no. Definition 2 specifies a joint law for the factual and alternative records. The reviewer's intuition is correct: Layer 2 quantity arises in an important special case. In a DAG, under Eq. (7) and the conditions of Proposition 2, the downstream distribution at a fixed revision agrees with the corresponding interventional distribution. In the loan example, the approval probability then equals .
Under a more general r-FAC rule, two applicants with the same factual ratio and revised ratio can have different probabilities of alternative approval because their factual decisions differed. The interventional distribution alone does not determine that pairing.
Does r-FAC satisfy SCM consistency? In general, no, but u-FAC does. Consistency requires whenever . Under a null revision, u-FAC evaluates the same downstream equations with the same factual disturbances, so it reproduces all affected values: almost surely (Proposition 1). Under r-FAC, downstream values may be resampled even when . For example, under Eq. (7), an applicant denied at ratio can receive approval on a fresh draw at that same ratio. Thus, this provides no evidence that keeping the ratio unchanged could have reversed the original decision.
Multivariate costs and the credit-line example (W4)
Can the cost of changing credit lines depend on total debt? Yes, but with an extension to Eq. (5). The discrepancy can already involve several variables and their interactions, but the local proposal for only uses and its pillow. On , this excludes the child . The potential extension depends on whether the relevant cost is determined by current or resulting debt.
-
Current debt . Suppose closing one credit line costs more for an applicant with a larger outstanding balance. A cost captures this directly. An extended proposal can then give less weight to more costly changes. This introduces no cycle because factual debt is already observed.
-
Resulting debt . If the cost depends on the balance after the change, one can first propose , generate downstream, and evaluate the joint cost . This cost can rank successful scenarios. If it should instead guide the initial proposal, one can use the expected cost under the downstream debt law for each candidate .
Which results carry the main theoretical contribution? (W2, Q1)
Theorem 1 and Propositions 4, 5 and 6.
Proposition 4 and Theorem 1 establish the identification limit for same-unit explanations. Proposition 4 expresses the success probability of each revision as an average over the disturbances compatible with the factual history. Theorem 1 proves that even exact knowledge of the causal graph, observed law, and revision law can leave the u-FAC diagnostic weights undetermined. The missing information is how the same unit’s responses under different revisions are related.
Proposition 5 gives an explicit identification functional for r-FAC. When the downstream resampling kernels are specified functionals of the observed law, Eqs. (13)–(14) show how to compose them in topological order and average over omitted factual inputs using . Lemma 1 justifies this averaging under the stated pillow and scope conditions. This result concerns identification under that law. Whether it represents same-unit responses is a separate modeling question.
Proposition 6 establishes why factual anchoring matters. A fixed marginal distribution of proposed revisions can yield different diagnostic weights for the same observed case. A case-specific explanation therefore requires specifying how alternative values are paired with factual values.
We thank the reviewer for pushing us to clarify the interpretation of resampling, the scope of our identification result, and the role of downstream costs. Their comments helped us explain these points more clearly in the revised manuscript.
Since the edits are expository and the theoretical results stand, we hope the clearer presentation will be reflected in an updated score. We welcome further discussion of any remaining questions.
Thanks for the response
Thank you for the response.
(W1) Adding the explicit diagnostic query is helpful, but the discretion argument is unconvincing: is already fully auditor-specified, and an auditor may legitimately want it to depend on (in the recourse setting, this would be the typical problem in fact; those with a positive decision will never be audited in a recourse setting). The remaining reason seems to be technical convenience.
Also, does the reduction of to hold under u-FAC? There the score is , which may depend on ?
(W2) The response still does not motivate restricting to . Non-descendant inputs beyond the pillow are not discussed, and factual descendants may be of interest, as discussed in (W1). Also, the framing concern remains unaddressed: I still cannot tell if this is a new notion of a counterfactual vs. a method for a specific explanation problem. Whichever it is, the semantics could be far clearer.
(W3) The authors confirm that r-FAC is neither interventional in general nor consistent. If r-FAC is neither an L2 nor an L3 object, why call it a counterfactual?
Based on the above (which I believe are significant concerns), I will maintain my score.
Followup Rebuttal to Reviewer ArHH
We thank the reviewer for the follow-up.
Outcome-dependent proposals and the success score (W1)
Can the auditor use the factual denial when proposing revisions? Yes. As noted in our first response, this is mathematically possible and, as the reviewer suggests, can indeed represent some types of recourse more directly when only denied applicants are audited. The extension can invalidate the independence in Lemma 1. For a denial, Bayes’ decomposition still applies with proposal factor . To accommodate this choice, Lemma 1 should condition on all factual inputs to the proposal, here . If is drawn using only these inputs and fresh randomization, it is conditionally independent of any remaining factual variables given .
We will add a new appendix section developing this extension and the adjustment to Lemma 1.
Moreover, the second paragraph of Section 3 was ambiguous, and our previous reply (mentioning target-success) added to the confusion. We intended as one example of how the analyst might resample under r-FAC. The general resampling law is , which may use the factual outcome and both the factual and revised Markov pillow values and . The law is an input and a designed choice. The analyst chooses which inputs to use, and the paper is written in its most general form: with all inputs used. That paragraph illustrates a simpler choice. It says "For example, drawing from ....". That is, drawing a fresh outcome from the observational law of given its parent, using the revised in place of .
We will clarify that Section 3 paragraph to not confuse it the target-success score introduced later in Section 5.
Why local inputs, and what problem does FAC solve? (W2)
Why restrict to ? This choice lets the auditor specify each revision locally: cases agreeing on these inputs receive the same proposal distribution at that node. We agree that a non-descendant outside the pillow, such as a baseline variable determining which revisions are feasible, can legitimately enter the rule. A factual descendant can also enter a retrospective proposal because its value is already known.
We will discuss both extensions, how they preserve Bayes’ decomposition, with omitted inputs handled as in W1.
What is the intended contribution? We favor the reviewer’s suggested framing. FAC addresses a specific diagnostic question: given this factual case, which proposed revisions are most likely among alternatives that achieve the target? The proposal determines how often each revision is considered. The downstream rule determines how often it succeeds.
We will foreground this question in the introduction and present u-FAC and r-FAC as two ways to answer it.
Is r-FAC counterfactual? (W3)
The reviewer raises a distinction that also arises in many topic in causal inference like principal strata and mediation analyses. On an augmented graph, r-FAC can be represented exactly as a sequence of stochastic interventions on the starred nodes [1]. With only single-world inputs, the resulting marginal outcome law is a Layer 2. When the resampler uses both factual (or ) and revised , the diagnostic query is cross-world, and thus it is Layer 3. For u-FAC, shared disturbances give the diagnostic query its same-unit interpretation, which is undisputed Layer 3 when the resampling law uses cross-world inputs.
The average treatment-effect on the treated (ATT) offers a familiar analogy, as the query evaluates an alternative outcome among units selected by factual treatment [2]. Mediation also illustrates why this distinction, vs , matters. The natural direct effect uses the same unit’s in , a Layer 3 quantity [3]. The interventional direct effect instead draws the mediator from a reference distribution, giving a Layer 2 interpretation and allowing identification under weaker conditions [4]. This mirrors u-FAC vs r-FAC!
Because _counterfactual_ can carry different meanings across frameworks, including SCMs and Rubin’s causal model, we will use the more precise SCM terms “Layer 2” and “Layer 3” to distinguish these queries in the final paper.
Thank you again!
References
[1] Correa, J., & Bareinboim, E. (2020). A Calculus for Stochastic Interventions: Causal Effect Identification and Surrogate Experiments. AAAI, 34(6), 10093–10100
[2] Shpitser, I., & Pearl, J. (2009). Effects of Treatment on the Treated: Identification and Generalization. UAI, 522–529
[3] Pearl, J. (2001). Direct and Indirect Effects. UAI, 411–420
[4] Vansteelandt, S., & Daniel, R. (2017). Interventional Effects for Mediation Analysis with Multiple Mediators. Epidemiology, 28, 258–265
Rebuttal to Reviewer ArHH: 1/2
We sincerely thank the reviewer for their careful assessment and for identifying where the motivating example needs more precision. We present our rebuttal below.
The motivating example and the role of the factual outcome (W1)
Why does the revision law does not condition on the factual outcome Y=y? We have made this clearer by strengthening the motivating example with the diagnostic query:
This diagnostic query does indeed conditions on the factual outcome because it is part of the case we seek to explain.
The revision law is an input to this query. For instance, on the graph Z→A→Y, the auditor can specify the revision ΓA⋆(⋅∣a,z) using the factual exposure and pre-outcome information (before learning the decision). This restriction gives the law a direct interpretation as a resampling rule and connects it to natural-value-dependent policies, which use the exposure that arose naturally to define its alternative.
Writing h=(z,a,0), Bayes’ rule gives Eq. (11) for discrete proposals:
\propto \Gamma_{A^\star}(a^\star\mid a,z)\, P(Y^\star=1\mid A^\star=a^\star,H=h).$$ **Question on first factor: could $\Gamma_{A^\star}$ also condition on $Y=y$?** Allowing $\Gamma_{A^\star}(a^\star\mid a,z,y)$ would introduce no cycle: the factual decision $Y$ is already realized when $A^\star$ is drawn, and revising $A^\star$ cannot change that earlier decision. Thus, Bayes’ rule still applies. The concern is how much discretion this gives the auditor. In the example, after seeing a denial, the auditor could give a particular debt-to-income ratio a higher proposal weight than they would after an approval. Even if two ratios have the same probability of success for this applicant, the more frequently proposed ratio will receive greater diagnostic weight. An outcome-dependent proposal can therefore strongly influence the explanation. Asking the auditor to specify the weights **among proposals that yield approval would go further and amount to supplying the explanation itself**. Moreover, if the proposal depends on the factual outcome, Bayes’ rule still applies, but Lemma 1’s implication $A^\star\perp Y\mid(Z,A)$ need no longer hold. We therefore favor our original modular proposal rule: the auditor can specify the revision, **before the decision**, using only the applicant’s history and observed ratio. As said before, the final diagnostic query still conditions on the factual denial. **Question on second factor: why does $P_{Y^\star}(\cdot\mid A^\star)$ appear?** The general success score in Eq. (11) would be $P(Y^\star=1\mid A^\star=a^\star,H=h)$. Since $A$ is the only parent of $Y$ **in the provided example**, the score reduces to $P(Y^\star=1\mid A=a^\star)$. This is why $P_{Y^\star}(\cdot\mid A^\star)$ appears in the example. We have revised the motivating example to clarify how the factual outcome enters the diagnostic query. ## Why restrict the kernel inputs, and how does u-FAC relate to soft interventions? (W2) **Local inputs.** We agree with the reviewer’s analogy. A soft intervention can assign a value stochastically, and a policy may use information beyond the node’s original parents. u-FAC likewise permits stochastic assignment, anchors it at the *observed* value, and evaluates downstream equations using the same unit’s factual disturbances. Proposition 2 makes this connection precise for fixed assignments and deterministic shifts of factual values. Following the reviewer questions, we now discuss broader inputs and two caveats. Conditioning on a *factual* descendant gives the auditor more discretion to shape the explanation and can invalidate Lemma 1’s anchoring independence. Conditioning on a *starred* descendant can create a cycle because its value may itself depend on the proposed revision. These considerations motivate our choice of modular revision laws that respect the topological order. **Why the exponential discrepancy/cost?** Equation (5) is optional. Its rationale is specific: if the analyst wants a revision law $q$ that trades off expected discrepancy/cost against departure from a reference law $Q$, minimizing $\mathrm{KL}(q|Q)+\beta E_q[\mathrm{cost}]$ yields $dq\propto e^{-\beta\,\mathrm{cost}}dQ$ (Appendix B, Lemma 2). Thus the exponential form follows from a penalized information-theory criterion. **This is the choice typically made for backtracking counterfactuals**