- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper discusses whether commonly used R-loss and doubly robust loss for conditional average treatment effect (CATE) remain reliable under hidden confounding. The authors construct an example to demonstrate that as hidden confounding strengthens, the population minimizers of R-loss and DR loss become biased with respect to the true CATE, while the validation losses decrease. They also validate this phenomenon with synthetic experiments.
Critical strengths
-
This paper explicitly constructs an example to demonstrate that hidden confounding can make CATE validation look better as causal error grows.
-
The extensive experiments validate the theoretical observation across eight different CATE estimators and 24 synthetic environments, suggesting that the phenomenon is not specific to a particular estimator or data-generating process.
Critical weaknesses
1.The main finding is not entirely unexpected. Under unobserved confounding, the standard identification formula for CATE becomes invalid. Since both R-loss and DR-loss rely on the observed-data distribution, it is natural that they may favor an associational target rather than the true causal effect. A closely related omitted-variable-bias mechanism has already been extensively studied in [1]. Although the paper demonstrates the more specific phenomenon that validation loss can decrease as causal error increases, it is unclear how much additional theoretical insight this construction provides beyond existing omitted-variable-bias theory.
- In real observational studies, neither the unobserved confounders nor the true treatment effects are typically available. Consequently, it is unclear how practitioners could implement the proposed framework on real datasets. The paper would benefit from more concrete guidance on how to diagnose in practice.
[1] Chernozhukov, V., Cinelli, C., Newey, W., Sharma, A., and Syrgkanis, V. (2022). Long Story Short: Omitted Variable Bias in Causal Machine Learning. NBER Working Paper No. 30302. https://arxiv.org/abs/2112.13398
Questions for authors
Please see the weakness section.
4 comments
The clarification that the proposed audit is intended for development-time evaluation is helpful. However, in simulator-based evaluations where the true CATE is available, researchers can already assess estimator performance directly using the oracle group-truth CATE. I still have the following questions:
-
Why is it meaningful to compare the numerical values of validation losses across different data-generating distributions? R-loss and DR loss are primarily designed to compare estimators under a fixed distribution. Their absolute values depend on distribution-specific quantities, including conditional outcome variance and propensity scores. Consequently, it is not clear why decreasing validation loss across different distributions should be interpreted as improving causal estimation accuracy in the first place.
-
If the goal is to study robustness to hidden confounding, why not directly measure the sensitivity of each estimator's causal error to confounding severity? In the authors' simulation framework, the true CATE is available. They could evaluate how rapidly each estimator's PEHE deteriorates as confounding increases, and compare the resulting sensitivity curves. This seems more directly informative about robustness than comparing absolute validation losses across distributions.
Why is it meaningful to compare the numerical values of validation losses across different data-generating distributions?
Comparing validation losses along a controlled shift tells us what feedback practitioners would receive as causal accuracy changes. Consider a hospital that refits a CATE estimator after new clinicians begin relying more on an unrecorded frailty assessment when assigning treatment. The team can observe the held-out loss but cannot observe the true treatment effects. A falling loss could reassure them just when there is reason to revisit their assumptions about treatment assignment.
We simulate this scenario by strengthening the influence of an unobserved factor on treatment assignment while holding covariates and potential outcomes fixed. Across 24 structurally varied environments, the pattern appears at intermediate severities as well as at the endpoints. As suggested, we track each estimator’s oracle root-PEHE to measure how much accuracy it loses. Tracking R-loss and DR loss alongside it shows whether the observable feedback reflects that deterioration. Chernozhukov et al. bound the omitted-variable bias, leaving open the direction of validation losses. That direction matters in the hospital example because falling R-loss or DR loss could reassure clinicians even as CATE estimates worsen.
On your second question, we use oracle root-PEHE to measure robustness in simulation, and the hospital example illustrates why we also track validation loss in settings where practitioners cannot compute PEHE because the true CATE is unknown.
Regarding "Consequently, it is not clear why decreasing validation loss across different distributions should be interpreted as improving causal estimation accuracy in the first place.
We agree that a decrease in loss alone does not establish an improvement in causal accuracy. Your question prompted a related check. Can the validation loss at baseline help identify which estimator will be most accurate after confounding increases?
Within each pool and simulated environment, we compared the estimator with the lowest mean validation loss at severity zero with the estimator with the lowest mean root-PEHE at severity one, averaging over 20 seeds. The two matched in 91.7% of EconML environments (22/24) but only in 66.7% of neural environments (16/24). The counts were the same for R-loss and DR loss.
These results suggest that baseline validation can help choose among candidates under a specified shift, even when a subsequent decrease in loss gives misleading feedback about whether their causal accuracy improved. Future work could test whether baseline scores reliably guide that choice across other shifts. Understanding the gap we observed between the EconML and neural pools could also shed light on which modeling and training choices help baseline scores remain informative as hidden confounding increases.
What the criterion-level results add beyond omitted-variable bias, and what practitioners can do
We thank the reviewer for recognizing the paper’s soundness and empirical breadth. We have uploaded a revised paper and supplementary ZIP that incorporate the new items summarized below. The revision clarifies that our contribution is not the existence of hidden-confounding bias, but that standard validation criteria can systematically reward it.
1. What is added beyond omitted-variable-bias theory?
Chernozhukov et al. bound omitted-variable bias for average causal parameters using the extent to which a latent variable explains treatment and outcome. Our complementary question is criterion-level: once hidden confounding displaces the causal target, what feedback does a held-out CATE validation loss provide?
Let τ(x) denote the CATE, gR(x) the observed treated–untreated conditional outcome difference, and B(x)=gR(x)−τ(x). Revised Section 3 and Appendix A establish
LR(τ)−LR(gR)=E[e(X)(1−e(X))B(X)2]
and
LDR(τ)−LDR(gR)=E[B(X)2].
Thus omitted-variable bias becomes a validation advantage: whenever B=0, both criteria prefer the confounded observational target to the true CATE. They share that target but impose different geometries—propensity-weighted for R-loss and unweighted for DR loss.
The result extends beyond the balanced illustrative construction. For an additive latent outcome model and arbitrary propensity, Appendix A shows that, along paths holding the outcome model and distribution of (X,U) fixed, the R-loss of the true CATE is invariant to the assignment mechanism. The attainable loss equals that constant minus E[e(1−e)B2]. Each unit of overlap-weighted squared bias is therefore one unit of apparent validation improvement.
These identities are short consequences of the standard R-loss decomposition; our claim is that their criterion-level consequence, to our knowledge, had not been articulated and tested as a validation failure. Omitted-variable-bias theory also does not address the distinction between relative selection and absolute monitoring. In our base families, selection regret remains only .006–.013, while the oracle-best candidate’s root-PEHE rises by approximately 1.05. A criterion can therefore select a near-best member of its pool while providing increasingly favorable feedback about a pool whose absolute causal accuracy is deteriorating.
The experiments establish that this mechanism appears at practical sample sizes with estimated nuisances, regularization, and heterogeneous learners. Across four EconML and four neural estimators, reversal occurs in 92–100% of paired hidden-confounding comparisons. Mechanistic controls in the EconML pool further show that systematic reversal disappears when either U→T or U→Y is absent. Under unconfounded sample-size degradation, causal error and both losses instead rise together.
We revised Section 2 to position Chernozhukov et al. as complementary sensitivity analysis. The introduction and Section 3 now credit the standard decomposition while identifying the criterion-gap, assignment-path, and selector-versus-monitor results as our contribution.
2. What can practitioners and model developers do?
We agree that one observational dataset cannot reveal its hidden confounders or oracle CATE. The proposed audit is therefore not a post hoc detector certifying that an estimate is unconfounded. Proposition 1 explains why: both losses are minimized by the observed conditional outcome contrast, so the losses alone cannot distinguish it from the CATE without additional structure such as credible proxies, negative controls, or sensitivity assumptions.
There is nevertheless an immediate practical implication: a lower held-out R-loss or DR loss across sites, time periods, or model revisions is not by itself evidence of improved causal accuracy. It can accompany causal deterioration when the identification environment changes.
The severity-path audit is primarily a development-time protocol:
The path can also support model development. An exposed failure mode defines a synthetic task family that could augment pretraining for a prior-fitted causal estimator such as CausalPFN; we leave this audit-guided adaptation to future work. For applied analyses without oracle outcomes, sensitivity analysis remains necessary. Our audit instead tests whether validation remains aligned with causal accuracy under plausible violations. Section 6 gives its scope, calibration anchors, and controls.