- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks an important question: Which observed covariates are responsible for the confounding bias in an observational causal analysis? The authors argue that conventional SHAP applied to propensity-score models, outcome models, or CATE models does not directly answer this question. For example, a variable can be highly predictive of treatment but be an instrument rather than a confounder, while a genuine confounder can have little importance for CATE heterogeneity because its effects on the two potential outcomes cancel. I consider the conceptual contribution novel and meaningful.
Critical strengths
- Clear distinction between different notions of feature importance. This is much stronger than simply applying standard SHAP to a causal ML model and claiming that the result explains confounding.
- The Shapley formulation is mathematically elegant. Because the full adjustment set has zero residual bias, the Shapley values decompose the bias associated with moving from no adjustment to full adjustment. This gives the attribution a principled accounting interpretation. In particular, the paper is not merely taking an arbitrary predictive score and putting SHAP on top of it. The authors deliberately define the causal quantity first, then use Shapley decomposition to explain that quantity. This is a methodological strength.\
- The distinction from nuisance-model SHAP is convincing. The paper derives an alternative expression showing that residual bias depends on conditional covariance between treatment assignment and potential-outcome regressions. This is useful because it explains why independently explaining the propensity model or outcome model cannot reproduce the proposed attribution.
Critical weaknesses
- The paper may be overclaiming what it measures. This is my biggest concern. The method does not directly identify “confounders” in the causal-graph sense. What it actually measures is closer to the contribution of observed covariates to residual bias relative to the selected full covariate set. The authors themselves acknowledge this in the limitations: if important confounders are unobserved, the attribution is only relative to the observed reference set.
- Dependence on the full adjustment set is fundamental. The entire method assumes that the full is a valid adjustment set. The paper explicitly defines the full-covariate CATE as the reference causal quantity. But this creates a potentially serious problem. It is bias relative to an estimated reference adjustment strategy.
- Shapley attribution does not equal causal responsibility. Shapley values are excellent for decomposing a cooperative game, but the resulting phi_j should not automatically be interpreted as the causal contribution of X_j to confounding. Consider highly correlated variables: X_1 \approx X_2. Both may represent the same underlying latent factor. Shapley symmetry will tend to distribute attribution between them. But this does not mean that each variable independently causes the confounding. It means that the two variables share the ability to reduce the bias under the chosen game.
Questions for authors
- What happens if the full covariate set contains a collider?
- How should proxy confounders be interpreted?
- Why use signed bias as the primary notion of “strength”?
3 comments
I have read the rebuttals by the authors, the authors have presented substantial refinement for their paper, and addressed several of my concerns. after careful consideration, I will keep my score unchanged.
Thank you for taking the time to read our rebuttal and for acknowledging the improvements to our paper. Please feel free to share any concerns that remain open, and we will be happy to address them.
Thank you for the thoughtful feedback and for finding our conceptual contribution novel and meaningful and our Shapley formulation principled. We addressed all of your points (changes in blue). Your comments also motivated us to extend ConfoundingSHAP to Shapley interactions, and we now validate it against an exact oracle game.
Response to weaknesses
W1: What the method measures
We agree that this distinction is important. In our formulation, ϕj is the average change in the signed residual bias when Xj is added to an adjustment set, and we now state explicitly that it is not a label for the role of Xj in the causal graph. We also added two results on this: covariates jointly independent of (A,X−j) receive exactly zero attribution, whereas an instrument can receive a nonzero one (bias amplification).
Action: We made this explicit throughout our entire revised paper. In our new Supplement C, we added the definition, the above condition, and an instrument example.
W2: Dependence on the full adjustment set
A valid reference set is the standard assumption in adjustment-based effect estimation, and our goal is to explain confounding under this assumption. With an insufficient reference set, the attributions sum to the unadjusted contrast minus the full-adjustment contrast. Sensitivity analysis can assess bias in the reference contrast (Eq. 5). At n = 5000, the end-to-end normalized error against the oracle is 0.07 for 11 covariates and 0.13 for 25.
Action: For space reasons, the connection to sensitivity analysis was previously in the appendix. We moved it to the main paper (see Sec. 4) and added the oracle-based evaluation (see our new Supplement E).
W3: Shapley attribution and causal responsibility
We agree that ϕj should not be read as causal responsibility, and the equal split among covariates with the same information is a property of the Shapley value itself. Your example motivated us to extend our method to pairwise Shapley interactions, which quantify how the contribution of one covariate changes once another one is adjusted for. For two noisy proxies of one latent confounder, each proxy removes 0.44 of the bias on its own but only 0.09 once the other is adjusted for. Hence, each receives 0.27, and the interaction I12=−0.34 shows this overlap. We are careful with the interpretation: interactions describe the dependence on the adjustment context but not where it comes from (e.g., we obtain a comparable value for two independent confounders under non-linear treatment assignment).
Action: We added the extension and its interpretation in our new Supplement C and refer to it in Sec. 4.
Response to questions
Q1: Collider in the full set
A reference set containing a collider can remain valid if it blocks all relevant noncausal paths. In our M-structure example, the collider receives −β/3, where β is the average bias from adjusting for the collider alone. If the collider path remains open under full adjustment, the reference contrast can be biased, and the attributions are still defined relative to that contrast.
Action: We added both cases with an example (in our new Supplement C) and state the assumptions explicitly in Sec. 3.
Q2: Proxy confounders
A proxy receives credit for the bias that is removed by adjusting for it, and several proxies of the same confounder share this credit. The credit becomes smaller when the proxies are noisier: each proxy receives 0.30 for exact copies and 0.27 when 5% of the values are misrecorded. The bias that imperfect proxies cannot remove is not attributed and is left to sensitivity analysis (see Sec. 4).
Action: We added this example (see our new Supplement C).
Q3: Signed bias
The signed bias keeps the direction: the global marginal contribution is the adjusted contrast before adding Xj minus the adjusted contrast after adding it. Hence, ϕj shows how strongly and in which direction omitting Xj shifts the estimate, and the attributions sum up to the signed confounding bias. Taking absolute values of stratum-level biases before averaging prevents cancellation between strata and discards direction. Cancellation can still occur within strata or across coalitions.
Action: We added the motivation in Supplement C.
Thank you again for the careful reading, which helped us to state more precisely what our attributions measure.