- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper addresses a relevant (yet easily overseen) confounding in assessing treatment effects. The work, coined CFSC, explains a workflow to correct a baised outcome profile using values from a randomized control trial. Through a shadow candidate source this bias is estimated and subtracted from the biased results. The results show that CFSC performs consistent accross different biases in synthetic data. In real data the results are comparable with state of the art or better dependant on the dataset.
Critical strengths
S1: Using the candidate from a randomized control trial is an interesting approach to correct observational studies that might be biased. The techniques used screens for adequate candidates that could contribute to the recovered target effect. The mathematics and the methodology seems sound.
S2: Results show promise and above all consistency.
Critical weaknesses
W1: The explanation on the selection of a shadow candidate is very vague. The work mentions, to remove candidates that add little outcome information, but does not provide a quantitative measure on how to do this. There is also no explanation on how large this shadow candidate should be. Can it be multiple dimensions?
W2: There is little information about the real world datasets. Little insight is given why CFSC workst better in one dataset but is comparable for the other dataset.
Questions for authors
-
How should we quantify a possible candidate for the Z (shadow candidate)? Can we use multiple shadow candidates and will het provide a better bias estimation?
-
What are your insights on why CFCS improves in JTPA dataset but is comparable to the state-of-the-art in the ACTG dataset.
3 comments
Q2. How do the JTPA and ACTG results differ?
Thank you for raising this question. In Table 3, CFSC and Causal-ICM have similar average treatment effect (ATE) errors and treatment-benefit ranking scores on ACTG. On JTPA, the difference is clearest in the ranking scores. CFSC has a positive mean AUTOC, while Causal-ICM has a negative mean AUTOC.
For average treatment effect error (), one possible explanation is how much the baseline covariates tell us about who enters the trial. In the JTPA benchmark, age affects the probability of entering the trial but is omitted from the prediction model. If an age group that benefits more from a training offer is underrepresented in the trial, the trial average can underestimate the target population’s earnings gain. CFSC uses prior earnings to estimate what people in the target population would earn with and without a training offer. This can reduce the error if prior earnings capture some of the difference in training benefit between the two populations. In ACTG, the baseline covariates already capture part of the treatment history used to sample trial participants, which may leave less room for baseline CD4 to improve the estimated average effect.
For treatment-benefit ranking (AUTOC), a correction matters only if it changes who ranks ahead of whom. We compared CFSC with RHC because both start from the same observational estimate. In JTPA, the two methods often prioritized different workers. Those selected only by CFSC had higher prior earnings on average than those selected only by RHC in every split (Appendix A.10, Figures 4 and 5). At the cutoff used for that comparison, the held-out randomized data gave a higher estimated earnings gain on average for the group selected by CFSC. At the more restrictive cutoffs we examined, the difference favored RHC (Figure 7). In ACTG, baseline CD4 helps predict follow-up CD4, but prioritizing patients requires predicting how much more each patient would benefit from one regimen than from the other. The two methods selected largely the same patients, and the group selected by CFSC showed no consistent increase in estimated treatment benefit (Appendix A.10, Figures 4 and 6).
Thank you answering the questions. They offered some clarity in total explanation.
Q1. How is the shadow variable selected, and can several candidates be used together?
We apologize for leaving the shadow screening procedure unclear. We consider pretreatment variables recorded in both sources, such as baseline measurements or prior outcomes. Variables prespecified to define the target effect stay in the adjustment set. We also exclude any variable used to construct source membership. We assess the remaining candidates in two steps.
Outcome relevance. Does the candidate carry information about the outcome beyond the adjustment covariates in both treatment arms? In the screened simulations, for example, we regress the candidate and outcome separately on the covariates in each trial arm and measure the absolute correlation between their residuals. In ACTG and JTPA, we check the adjusted outcome association on the design sample and require sufficient outcome information within every source-by-arm group.
Source exclusion. For each arm a, we ask whether the recruitment process makes it plausible that the candidate has the same distribution in both sources given the covariates and potential outcome Y(a). In ACTG and JTPA, we also check the candidate's residual association with source after adjustment for covariates, treatment and the observed outcome. A large residual association rules out a candidate under our screening rule. A small one leaves exclusion given Y(a) as a design assumption.
If several candidates qualify in the simulations, we select the one with the lowest held-out score, which measures both agreement between two estimates of the observational shadow mean and the size of the shift from the trial outcome regression to the recovered target mean. In ACTG and JTPA, we select the candidate with the strongest outcome information in its weakest source-by-arm group at the chosen regression degree. We use the sole qualifying candidate when only one passes and report a screening failure when none passes.
Each experiment used one scalar shadow. A prespecified vector can also be used if its components jointly satisfy exclusion and the vector carries outcome information in each arm. Complementary candidates could improve precision, but we did not compare single and multiple shadows empirically. Appendix A.7.1 now gives the candidate pools, screening pseudocode, thresholds and examples of how baseline CD4 and prior earnings were selected (Table 4 and Algorithm 2).