- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes AO-CES, a diagnostic for hidden confounding of a specified treatment–outcome relation using pretreatment proxy measurements. It predicts T from contexts alone and again with W added; the difference, called the anchor, is then tested on held-out data for association with T beyond contexts and with Y beyond T and outcome contexts. A correction term removes the leading estimation bias introduced by the fitted anchor, and Theorem 1 gives an asymptotic false-positive guarantee under learning-rate conditions, while Proposition 1 characterizes when proxy mixtures create apparent confounding with no shared cause. Experiments cover 21 simulation conditions, a semi-synthetic use of WDBC covariates, a matched binary-treatment study, and an application that annotates FCI causal-discovery output with proxy-based confounding evidence under familywise-error control.
Critical strengths
1、The method achieves strong empirical performance across the reported simulation conditions and the FCI application.
2、The FCI application is sensible packaging that uses causal-discovery output to select candidates and adds confounding annotations with familywise-error control while reporting both the screening and multiplicity penalties.
Critical weaknesses
-
All validation is synthetic or semi-synthetic, and the experiments repeat one narrow task across all 21 conditions rather than covering diverse tasks or any real observational study.
-
The feature-learning comparison uses no variant newer than 2017, and binary-treatment power is near zero at n equals 800 and only 29 to 37 percent at n equals 3,200.
Questions for authors
-
Proposition 1 and Figure 2a show that proxy mixtures violate the ideal outcome null even with no shared cause, and that the correction cannot repair this. Is there any data-driven check that flags when the proxy set itself violates the measurement conditions, or must a user simply assume validity?
-
Is there any real observational dataset, with a plausible proxy and a ground truth or at least a benchmarked effect, on which AO-CES produces a sensible answer?
-
Given 0 to 3 percent binary detection at n equals 800, what sample size would a practitioner need, and is there anything in the framework that can improve binary power beyond the prespecified logistic follow-up?
-
In the FCI application, overall recall is 33.3 percent at low noise and 10.0 percent at high noise while precision is 99.3 percent, and the abstract reports only the 93 to 100 percent conditional recall. Can you reconcile these figures so that the practical recall of the pipeline is stated as prominently as its precision?
4 comments
W2: Is there a comparison with a method newer than 2017?
Yes. We evaluated piecewise-linear encoding adapted from Gorishniy et al. (NeurIPS 2022). In the nonlinear-additive design at , corrected AO-CES achieved 70% power with this encoding and 67% with raw proxies. The three-point difference was uncertain (95% interval −2 to 8). Thus the comparison includes a recent encoding, without establishing that it outperforms raw proxies.
References: Gorishniy et al. (2022), NeurIPS; Taubman et al. (2014), Science; Oregon replication data.
Note the revised version will include all of this information, in an Appendix
Thank you for the detailed replies and the Oregon analysis. I read them carefully. The Oregon result is a useful illustration, but as you write yourselves it has no ground-truth label, so it does not really change my main concerns, which are the following.
My biggest concern is still that the experiments are one task with different settings. The 21 conditions are essentially the same procedure with switched parameters, so the diversity of what is actually tested is limited. I know the number of experiments alone does not decide novelty, but to me the experimental methods and the tasks are too simple. I also notice my W1 was only half answered in the rebuttal: the Oregon analysis covers the part about real observational data, but the part about one narrow task repeated across all 21 conditions was not addressed. Many methods discussed in the related work never appear in the experiments. I would add at least some of them as baselines, for example sensitivity-type checks such as Oster's coefficient stability (JBES 2019) or the robustness value of Cinelli and Hazlett (Political Analysis 2020), Rosenbaum bounds, negative-control-outcome tests in the sense of Lipsitch et al. (2010) and the proxy line of Miao et al. (Biometrika 2018), and CI-test based confounding detection like Entner et al. (2013). Without any of these comparisons it is hard for me to see where AO-CES stands among the tools people already use.
Second, the exposition of the method is confusing. The paper puts most of its weight on the task and on when existing approaches fail, while the core methodological contribution itself is not stated clearly. After reading it, it is still not easy for me to say in one sentence what the real innovation of AO-CES is. A paragraph that states this plainly would help the paper a lot.
Third, please add figures. A schematic of the anchor construction and of how the correction enters the test would help reviewers much more than another half page of formulas. In the AI era text is cheap, one clear figure is worth a lot.
Last small point, and this is only my personal impression: the style of subsection titles like 3.1, 3.2, 4.1 reads to me like AI-organized writing. Maybe it is just my preference, but together with the writing style of the text it made the paper feel less carefully prepared.
For these reasons I keep my score.
Thank you for clarifying the remaining concerns. We have revised the comparisons and presentation to address them directly.
1. Comparisons and broader evaluation. We added negative-control regression, a two-proxy bridge, Oster's coefficient stability, Cinelli–Hazlett robustness values, Rosenbaum bounds, and an Entner R1 adjustment-witness check, preserving their different targets and assumptions.
In 400 paired datasets per confounding condition, AO-CES and the designated negative-control test both achieved 100% detection, with 5% versus 6% null rejection. AO-CES received measurements without designated control roles.
We also evaluated adjustment and bias prioritization. Recording and adjusting for the omitted common cause removed AO-CES evidence in 393/400 datasets. Across 100 IHDP semi-synthetic replications, diagnostic strength tracked known selection bias (, 95% interval [.61, .79]), calculated from potential-outcome means. These replications reuse a cohort; this is benchmark evidence, not independent clinical validation.
2. Methodological contribution. AO-CES corrects the leading false outcome association introduced by estimating a treatment-predictive proxy score, with a validity guarantee under explicit measurement and learning conditions. Section 3 now states this directly.
3. Figures. Figure 2 shows anchor construction and where the correction enters; Figure 3 displays the empirical gains.
4. Presentation. We simplified the subsection titles and clarified the method explanation. Manuscript changes are marked in red; complete comparisons appear in Section 5 and Appendices M–O.
We would appreciate your reassessment in light of these comparisons, practical evaluations, and presentation changes.
Thank you for your comments and feedback.
Q1: Can data reveal invalid proxies?
Yes, some violations can be detected, but passing a check does not verify Assumption 2. Note that four linear measurements of one factor with mutually independent errors independent of that factor corresponds to σ12σ34=σ13σ24, where σij=Cov(Wi,Wj). Rejecting this covariance constraint provides evidence against that measurement model.
Passing can miss outcome information hidden in one measurement. For example, let all latent variables and errors be independent standard Gaussians:
T=U+εT,Y=βT+V+εY,W1=U+V+η1,Wj=U+ηj(∀j=2,3,4).
All off-diagonal proxy covariances equal one, so the constraints hold exactly. Yet A0=(W1+2W2+2W3+2W4)/9 gives a nonzero outcome moment, θY,1=1/9, despite no hidden common cause.
We tested this check in 400 simulated datasets per setting, using 200 observations for checking and 800 others for AO-CES. With two strongly mixed measurements, 389 datasets failed the check. Withholding conclusions for them reduced false annotations from 241 to 6, but 6of the 11 passing datasets still produced false annotations. Thus the check cannot replace justification of how the measurements were collected.
Q2 and W1: Is there a real-data example with an external benchmark?
Yes: the Oregon Medicaid lottery provides randomized assignment alongside observational enrollment. We analyzed 22,188 households, holding out 4,438. The observational treatment was enrollment during days 1–180, and the outcome was emergency-department use during days 181–360. Five pre-lottery utilization indicators served as candidate proxies for health and care-seeking behavior, with demographic and lottery covariates adjusted for separately.
AO-CES rejected for enrollment (p=.002) but not randomized offer (p=.830); six further splits of the same records gave the same pattern. Taubman et al. (2014) supply an external randomized effect benchmark: approximately seven percentage points more emergency-department use over a different 18-month window.
This benchmark does not provide a ground-truth confounding label. Prior utilization may directly influence later use, so proxy validity still needs justification. Lottery non-rejection mainly reflects the absence of proxy information about randomized assignment, not a separate validation of the outcome test.
Q3: What sample size is needed, and how can binary power improve?
Sample size. In this simulation, 6,400 total observations, with 20% held out, was the first tested size that gave over 80% power for both confounding signs. With the improved fits described below, power was 97% for positive confounding and 91% for negative confounding. At n=800, it remained 13% and 10%. These are design-specific results, not a universal sample-size requirement.
Improvement beyond logistic prediction. Keeping the logistic treatment predictor and relevance gate fixed, we simplified the downstream regressions to treatment-group means and main-effects ridge. At n=3,200, power rose from 55% to 76% for positive confounding and from 37% to 67% for negative confounding (100 datasets per sign). The paired gains were 21 and 30%points, with 95% intervals of 12–30 and 21–40. These matched follow-up results are separate from the original experiment cited in the review.
At n=800, the change reduced score standard deviations from about .20 to .08 while barely changing mean signals, explaining the power gain. At n=3,200, false-positive rates were 0–4% across the three unconfounded settings, at nominal 5%. A fresh treatment-only null check at n=6,400 gave 21/400 false positives.
Q4: Which FCI recall figures describe the full pipeline?
For the randomized graphs, full-pipeline recall is 33.3% at low noise and 10.0% at high noise. FCI retained 200 of 409 true confounded pairs. AO-CES detected 136 and 41 of those 200, respectively. Thus recall over all true targets is 136/409 and 41/409; precision is 99.3% and 97.6%.
The 93–100% figures come from separate structured graphs where FCI retained every target. The revised version will report the randomized graphs’ full-pipeline recall beside precision in the abstract and results, clearly distinguishing the benchmarks.