acceptodds
Under review as a conference paper at ICLR 2027

Beyond Ignorable Selection: Causal Fusion with Shadow Calibration

Abstract

Randomized controlled trials (RCTs) provide credible causal evidence but are often limited in size and representativeness, whereas observational studies (OBS) offer broader coverage but may suffer from hidden treatment confounding. Many data-fusion approaches that combine randomized and observational data rely on sample-selection ignorability or require treatment effects to be transportable across sources. We study data fusion when source selection is nonignorable, so source membership may remain associated with potential outcomes after conditioning on observed covariates. We introduce Causal Fusion with Shadow Calibration (CFSC), which uses shadow-variable restrictions to recover target-population outcome laws and calibrate randomized supervision for estimating treatment effects in the OBS target population. CFSC combines the observational treatment–outcome structure with a structured correction supervised by the recovered target-specific RCT signal. Under the stated conditions, we establish identification for target conditional and average treatment effects. In controlled simulations, CFSC remains stable as source selection strengthens and outperforms conventional fusion methods when both biases coexist. Constructed real-data benchmarks also show remarkable mean ATE-reference agreement and treatment-benefit ranking.

Primary area:
causal reasoning
Keywords:
causal inference, data fusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F3G8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper addresses a relevant (yet easily overseen) confounding in assessing treatment effects. The work, coined CFSC, explains a workflow to correct a baised outcome profile using values from a randomized control trial. Through a shadow candidate source this bias is estimated and subtracted from the biased results. The results show that CFSC performs consistent accross different biases in synthetic data. In real data the results are comparable with state of the art or better dependant on the dataset.

Critical strengths

S1: Using the candidate from a randomized control trial is an interesting approach to correct observational studies that might be biased. The techniques used screens for adequate candidates that could contribute to the recovered target effect. The mathematics and the methodology seems sound.

S2: Results show promise and above all consistency.

Critical weaknesses

W1: The explanation on the selection of a shadow candidate is very vague. The work mentions, to remove candidates that add little outcome information, but does not provide a quantitative measure on how to do this. There is also no explanation on how large this shadow candidate should be. Can it be multiple dimensions?

W2: There is little information about the real world datasets. Little insight is given why CFSC workst better in one dataset but is comparable for the other dataset.

Questions for authors
  1. How should we quantify a possible candidate for the Z (shadow candidate)? Can we use multiple shadow candidates and will het provide a better bias estimation?

  2. What are your insights on why CFCS improves in JTPA dataset but is comparable to the state-of-the-art in the ACTG dataset.

3 comments
AuthorsOpenReview ↗

Q1. How is the shadow variable selected, and can several candidates be used together?

We apologize for leaving the shadow screening procedure unclear. We consider pretreatment variables recorded in both sources, such as baseline measurements or prior outcomes. Variables prespecified to define the target effect stay in the adjustment set. We also exclude any variable used to construct source membership. We assess the remaining candidates in two steps.

  1. Outcome relevance. Does the candidate carry information about the outcome beyond the adjustment covariates in both treatment arms? In the screened simulations, for example, we regress the candidate and outcome separately on the covariates in each trial arm and measure the absolute correlation between their residuals. In ACTG and JTPA, we check the adjusted outcome association on the design sample and require sufficient outcome information within every source-by-arm group.

  2. Source exclusion. For each arm aa, we ask whether the recruitment process makes it plausible that the candidate has the same distribution in both sources given the covariates and potential outcome Y(a)Y(a). In ACTG and JTPA, we also check the candidate's residual association with source after adjustment for covariates, treatment and the observed outcome. A large residual association rules out a candidate under our screening rule. A small one leaves exclusion given Y(a)Y(a) as a design assumption.

If several candidates qualify in the simulations, we select the one with the lowest held-out score, which measures both agreement between two estimates of the observational shadow mean and the size of the shift from the trial outcome regression to the recovered target mean. In ACTG and JTPA, we select the candidate with the strongest outcome information in its weakest source-by-arm group at the chosen regression degree. We use the sole qualifying candidate when only one passes and report a screening failure when none passes.

Each experiment used one scalar shadow. A prespecified vector can also be used if its components jointly satisfy exclusion and the vector carries outcome information in each arm. Complementary candidates could improve precision, but we did not compare single and multiple shadows empirically. Appendix A.7.1 now gives the candidate pools, screening pseudocode, thresholds and examples of how baseline CD4 and prior earnings were selected (Table 4 and Algorithm 2).

AuthorsOpenReview ↗

Q2. How do the JTPA and ACTG results differ?

Thank you for raising this question. In Table 3, CFSC and Causal-ICM have similar average treatment effect (ATE) errors and treatment-benefit ranking scores on ACTG. On JTPA, the difference is clearest in the ranking scores. CFSC has a positive mean AUTOC, while Causal-ICM has a negative mean AUTOC.

For average treatment effect error (ϵATE\epsilon_{\mathrm{ATE}}), one possible explanation is how much the baseline covariates tell us about who enters the trial. In the JTPA benchmark, age affects the probability of entering the trial but is omitted from the prediction model. If an age group that benefits more from a training offer is underrepresented in the trial, the trial average can underestimate the target population’s earnings gain. CFSC uses prior earnings to estimate what people in the target population would earn with and without a training offer. This can reduce the error if prior earnings capture some of the difference in training benefit between the two populations. In ACTG, the baseline covariates already capture part of the treatment history used to sample trial participants, which may leave less room for baseline CD4 to improve the estimated average effect.

For treatment-benefit ranking (AUTOC), a correction matters only if it changes who ranks ahead of whom. We compared CFSC with RHC because both start from the same observational estimate. In JTPA, the two methods often prioritized different workers. Those selected only by CFSC had higher prior earnings on average than those selected only by RHC in every split (Appendix A.10, Figures 4 and 5). At the cutoff used for that comparison, the held-out randomized data gave a higher estimated earnings gain on average for the group selected by CFSC. At the more restrictive cutoffs we examined, the difference favored RHC (Figure 7). In ACTG, baseline CD4 helps predict follow-up CD4, but prioritizing patients requires predicting how much more each patient would benefit from one regimen than from the other. The two methods selected largely the same patients, and the group selected by CFSC showed no consistent increase in estimated treatment benefit (Appendix A.10, Figures 4 and 6).

Thank you answering the questions. They offered some clarity in total explanation.

Reviewer_vVGsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to combine randomized and observational data when observational treatment is confounded and trial participation is nonignorable. It uses a shadow variable to recover target-population outcome information, then uses the recovered effect to correct an observational estimator.

Critical strengths

The problem is important. Randomization within a trial does not make its participants representative of the target population. The paper clearly states the shadow-variable and completeness assumptions and gives an identification argument under those assumptions.

Critical weaknesses
  1. Assumption 2 is hard to justify in the motivating example. It requires conditional independence between Z and S on X and Y(a) for each arm. If motivation affects participation and both potential outcomes, and Z is a proxy for motivation, conditioning on either potential outcome alone may leave Z associated with participation. In the simulation, each potential outcome determines U given X, making exclusion hold by construction. The authors should explain when this restriction is plausible beyond that setting.
  2. The simulations cover a favorable setting. It is helpful to investigate what happens with additional outcome variation, weaker shadows, and misspecified recovery models. There is also a gap between the general method and what is tested. The main text develops density-ratio recovery through an inverse problem, but Appendix A.6.1 uses a reverse conditional-mean model that is linear in the outcome. This is a special case, but the results do not show how the general recovery procedure performs.
  3. The recovery and evaluation populations appear different in the real-data experiments. Appendix A.6.1 estimates the target shadow mean from the outcome-dependently thinned OBS sample, while evaluation uses intact target folds. Equation 31 needs the shadow mean for the population whose effect is being recovered. Thinning based on outcome can change this mean even conditional on X. Why does recovery from the thinned sample identify effects in the intact population? It is also unclear whether the shadow exclusion restriction survives the thinning.
  4. The screening experiment provides limited evidence for choosing a shadow variable. The candidates are one valid shadow and two independent noise variables, so screening mainly tests relevance. The paper acknowledges that screening cannot verify exclusion or completeness, but it would help to show how often it selects an invalid candidate and what happens afterward.
  5. The baseline comparisons could be fairer. Only CFSC uses Z. Giving baselines access to (X, Z) would help separate the benefit of this extra information from the benefit of shadow calibration. Several baselines also use linear outcome or effect models despite nonlinear effect heterogeneity in the simulation. Comparisons with matched model classes would be more informative. It would also be useful to compare with using the shadow-recovered effect directly, without the final correction stage, to assess whether combining it with the observational estimator improves accuracy.
Questions for authors
  1. Can the authors give a realistic example where selection exclusion holds for both arms?
  2. How does CFSC perform with additional outcome noise, a weak shadow, or a misspecified reverse regression?
  3. Which population does recovery identify after outcome-dependent thinning?
  4. How does CFSC compare with direct shadow-mean recovery and baselines given the same covariates?
5 comments
AuthorsOpenReview ↗

W1 / Q1. When can shadow exclusion hold in both arms?

Thank you for pointing out that if job-seeking motivation UU affects both trial participation and potential earnings in each arm, a motivation questionnaire may still predict enrollment SS given background characteristics XX and either potential outcome Y(a)Y(a). We agree. Consider machine operators who can volunteer to retrain as maintenance technicians. Two workers might earn the same wage without retraining, yet one may be more motivated than the other. If the more motivated worker is also more likely to volunteer, the questionnaire can still distinguish their willingness to enroll. As you note, Eq. (19) covers a special case in which XX and either potential outcome determine UU.

However, even when workers with the same wage differ in motivation, the same training program provides an example in which exclusion holds in both arms. Consider pre-training job grade GG, recorded as junior or senior in both payroll files, and take Z=GZ=G. Suppose workers can remain in their original job or retrain for a maintenance job, and that basic pay in each job follows a contractual wage schedule. If junior and senior workers earn different basic wages in each job at the same seniority and other pay determinants XX, knowing a worker’s grade helps predict their wage in that job. Conversely, once XX and the wage in that job are known, the wage schedule already tells us the worker’s grade. Two such workers may still differ in their willingness to volunteer, but learning their grade adds no further information about which worker is more likely to enroll. The same pre-training grade can therefore satisfy the shadow-variable conditions in both arms.

Formally, provided both grades occur in the trial at a given XX, the difference in pay between grades gives outcome relevance, G⊥̸Y(a)∣X,S=RCTG\not\perp Y(a)\mid X,S=\mathrm{RCT}. Each job’s wage schedule also gives a relation G=ga(X,Y(a))G=g_a(X,Y(a)), which implies selection exclusion, G⊥S∣X,Y(a)G\perp S\mid X,Y(a). Both conclusions hold separately for a=0,1a=0,1. Appendix A.1 (p. 13) gives the proof. If the two grades can earn the same wage at fixed XX in one job, that wage no longer determines grade, and exclusion in that arm needs a separate justification.

In practice, we also consider how a candidate was measured and how participants entered the study when assessing whether exclusion is plausible. In the held-out JTPA design sample, for example, we checked whether each candidate added information about follow-up earnings and whether it retained a substantial linear association with source membership after adjustment for covariates, treatment, and observed earnings. Within each treatment arm, we also checked whether the relationship between the candidate and earnings transferred across sources. These diagnostics supported the choice of prior earnings as the JTPA shadow variable. Arm-specific exclusion still requires justification from the study setting.

AuthorsOpenReview ↗

W2 / Q2. Sensitivity to outcome noise, weak shadows, and model misspecification

We thank the reviewer for pointing out that our original simulations used a favorable setting and for asking how robust CFSC is when those conditions change. In response, we conducted three additional simulation studies to assess how accurately CFSC estimates treatment effects in the target population as we increase outcome noise, weaken the shadow variable, or misspecify the reverse regression. The results of the three sensitivity analyses are reported in Appendix A.12, Tables 8–10.

When the added outcome noise preserved the shadow exclusion restriction, CFSC’s mean PEHE increased only modestly with the noise level and remained lower than that of both RHC and Integrative-R at every level tested. When we fitted a linear reverse regression to a nonlinear conditional mean, CFSC retained this advantage at all three levels of misspecification. Weakening the shadow variable had a larger effect on CFSC’s accuracy. Integrative-R had lower mean PEHE than CFSC both when the shadow was weakest and when strong independent outcome noise violated the exclusion restriction.

You also point out that the linear reverse conditional-mean model in Appendix A.7.1 is a special case of the general recovery procedure. We therefore solved a discretized, regularized density-ratio inverse problem on the same splits. Conditional-mean recovery had lower mean PEHE in the original setting, and density-ratio recovery had slightly lower mean PEHE when the reverse relationship included a cubic term. The differences were small in both settings. Appendix A.12 gives the implementation details and paired results.

AuthorsOpenReview ↗

W3 / Q3. Which population does recovery identify after outcome-dependent thinning?

Thank you for identifying the mismatch between the recovery and evaluation populations in our original implementation. After correction, CFSC recovers effects in the OBS population defined by the initial source split, before outcome-dependent thinning. We construct the ACTG and JTPA benchmarks in two steps. The first makes trial membership depend on a baseline variable, using prior antiretroviral treatment strata in ACTG and age in JTPA. This induces source selection and defines the remaining individuals as the OBS target population. The second step thins only the OBS training folds. Higher follow-up outcomes make treated records less likely to be retained and control records more likely to be retained, inducing a biased observational treatment comparison. The intact OBS evaluation folds preserve the original randomized assignments and provide a reference for the population defined in the first step.

The mismatch concerns the target shadow mean hO(x)h_O(x) used in Eq. (31). Let R=1R=1 denote retention in the second step. The original implementation estimated E[Z∣X=x,S=O,R=1]E[Z\mid X=x,S=O,R=1] from the retained records, whereas recovery for the intact OBS population requires hO(x)=E[Z∣X=x,S=O]h_O(x)=E[Z\mid X=x,S=O]. In JTPA, for example, thinning on follow-up earnings can change mean prior earnings even at fixed XX. The original input therefore did not guarantee recovery of the intact OBS effect. Interpreting the original estimate as an effect in the retained population would additionally require the arm-specific shadow relationship from the RCT to hold after thinning on observed treatment and outcome.

We corrected this input and reran CFSC on ACTG and JTPA. We now estimate hO(x)h_O(x) from baseline (X,Z)(X,Z) in the OBS training fold before thinning. The arm-specific reverse regressions use the RCT, the OBS effect model uses the retained labeled records, and the evaluation fold is held out from both fits. Under arm-specific exclusion, the specified reverse conditional-mean model, and the other identification conditions, including βa≠0\beta_a\ne0 in each arm, taking conditional expectations in the original OBS population gives hO(x)=ℓa(x)+βaμaO(x)h_O(x)=\ell_a(x)+\beta_a\mu_a^O(x). Equation (31) thus recovers the arm-specific mean potential outcomes in that population. Their difference gives its conditional average treatment effect, and averaging over its distribution of XX gives its population average treatment effect.Table 3 in Section 5.2 reports the updated CFSC results for ACTG and JTPA.

Reviewer_nkSeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on the problem of estimating causal effects in observational data, with access to data from a randomized trial. If causal effects are transportable (i.e., if the RCT population is itself a representative sample), then this problem is very straightforward. This paper focuses on a possible violation of that assumption, where unobserved confounders influence selection into the trial itself, and proposes to resolve this issue using "shadow variables", a technique from the missing data literature that requires a variable that (a) is related to the variable of interest, but (b) is conditionally independent of the missingness indicator given covariates and the unobserved variable of interest.

Critical strengths

None

Critical weaknesses

A few notable weaknesses that give me pause.

Lack of Clarity

The first two are related to a lack of clarity, an all-too-common problem in papers in my batch...

(W1) The first weakness is the prevalence of dense prose that is difficult to parse. I will give one example to illustrate the broader trend (line 096 of Section 2.1):

For transport, outcome regression, sampling weights, and calibration weighting adjust for observed population differences.

This is an example of a difficult-to-parse sentence that is overly terse. I believe the idea is to write that "Methods that attempt to transport trial effects use a variety of methods to adjust for observed population differences, such as outcome regression..."

(W2) The second weakness is also related to clarity around assumptions, and being upfront about them in a way that is easier to parse. As with any causal inference method, it is important to be upfront about what kind of assumptions are required. Under Assumption 2 (conditions for a shadow variable to be valid), it is stated (lines 203-204)

"We call a candidate satisfying these restrictions a shadow variable. We do not assume that one is known a priori; the Shadow Recovery component of Section 4 describes how candidates are screened from the observed covariates"

Emphasis added by myself. I took "we do not assume that one is known a priori" to mean that a weaker assumption is required (e.g., that one exists among all of XX, or similar), and that some procedure would be proposed to identify shadow variables among some set of covariates if one exists.

But when this procedure is described (Section 4.1), it is quite vague, saying that a fixed candidate set (separate from XX!) is required, and that they "remove candidates that add little outcome information beyond X" and "for the remaining candidates, we fit the recovery equation and assess held-out moment residuals and tilt stability" (which, as far as I could tell, where not defined anywhere). See lines 310-314.

It is only on line 316 and onwards that we get the perplexing walking-back of any substantive claim of correctness or utility of this approach. Rather, we get the disclaimers on line 315 that "we call this stage Shadow Recovery because the shadow restriction enables recovery under nonignorable source selection, not because the procedure recovers Z itself" (emphasis added), and then later it is stated that "the screening diagnostics assess empirical compatibility, not validity of the identifying restriction itself: they do not prove Z⊥S∣X,Y(a)Z \perp S | X, Y(a) or completeness. The theoretical guarantee with screening therefore remains conditional on selecting a valid candidate with probability tending to one" (see lines 331-334)

I don't particularly care if the paper claims to be able to identify shadow variables! It would be fine to just say "we assume a shadow variable ZZ is known"! But the confusing exposition leaves a bad taste in my mouth, as if the original version wanted to make the claim that these could be identified from data, and a later revision just decided to add all these qualifiers on top of a sub-method (screening for ZZ) that doesn't contribute anything substantive, instead of just dropping the screening component entirely.

Mismatch between Assumptions, Figure 1, experimental setup

(W3) Figure 1 gives a causal graph (in the third panel) that is supposed to represent the setup in the paper. However, the conditional independence Z⊥S∣X,YZ \perp S \mid X, Y does not actually hold in this graph (conditioning on YY opens the path S←U→Y←ZS \leftarrow U \rightarrow Y \leftarrow Z with YY as a collider), making it a poor representation of the type of graph that motivates this work.

I must stress that this is a pretty fatal error, in my opinion; Figure 1 should help illuminate the assumptions being made and why they are plausible, but instead shows a case in which the assumptions do not actually apply.

In addition, Figure 1 seems to bear no relation to the setup in the synthetic experiments, where ZZ is just a noisy copy of UU, and UU impacts sample selection. The only reason that (I think?) the synthetic experiment satisfies the stated conditions is that Y(a)Y(a) is a deterministic function of U,XU, X (see Equation 19!) and so if we condition on X,YX, Y, we can perfectly recover UU, and hence Z⊥S∣X,Y(a)Z \perp S \mid X, Y(a), but this seems like a very strange way to get this condition to hold.

Significance: Plausibility of assumptions?

(W4) I am still trying to wrap my head around what a plausible shadow variable would be in this scenario, given the trial / observational study motivation. Given that a core argument is that this idea of a shadow variable can be "transported" from the missing data literature to the observational/experimental data fusion literature, I feel it is incumbent upon the authors to at least give one or two examples where there is a variable that plausibly satisfies these conditions.

Questions for authors

I am asking these questions in case I missed something in the paper:

  • Does the procedure for selecting ZZ from a set come with any guarantees if not all members of the set are valid shadow variables? Or is this procedure only valid if every member of the set is already a valid shadow variable?
  • Am I missing something, or does the motivating causal graph in Figure 1(c) violate the assumption that ZZ is a shadow variable, since ZZ is not d-separated from SS in either the graph itself, or in the SWIG where we include Y(a)Y(a), since S←U→Y(a)←ZS \leftarrow U \rightarrow Y(a) \leftarrow Z, and Y(a)Y(a) is a collider?
  • Can you give an example of a plausible shadow variable in the setting where we are trying to combine trial data and observational data? Do not give examples from the traditional shadow-variable literature, since your argument is that nobody has applied the idea of a shadow variable to the trial/observational data setting.
4 comments
AuthorsOpenReview ↗

W1. Clarity of the exposition

Thank you for raising this issue and for pointing to a specific example. We agree that clear writing is essential for readers to follow the argument. In trying to stay within the page limit, we condensed some explanations too much, which made parts of the paper harder to read. We have since read through the full manuscript and revised passages where the connections between ideas needed to be clearer.

AuthorsOpenReview ↗

W2 / Q1. Assumptions and guarantees for shadow screening

Thank you for pointing out the ambiguity in the passage after Assumption 2. We wrote that a valid shadow variable need not be known in advance and immediately referred to screening. Read together, these statements suggest that screening will find a valid shadow whenever one is present. We meant that researchers can prespecify a set of pretreatment variables measured in both the trial and the observational study. The set may include invalid candidates, and researchers need not know in advance which candidate is valid.

We acknowledge that consistency after screening requires an additional assumption. As the screening sample grows, the probability of selecting a valid shadow variable must tend to one. This was the condition in Eq. (27) of the original manuscript. We now state it as Assumption 3 in Section 3.2, immediately after the passage on screening that you highlighted. Corollary 2 in Appendix A.5 cites it directly. Under this assumption and the other regularity conditions, with independent samples for screening and estimation, the screened CATE estimator is uniformly consistent on the common support, and the screened ATE estimator is consistent for the target population.

Algorithm 2 in Appendix A.7.1 gives the screening rule used in the experiments. In the simulations, we first check each candidate’s association with the outcome after adjusting for XX within each trial arm. For candidates that pass, we compare estimates of the candidate’s conditional mean fitted on two OBS subsamples. We also compare the recovered outcome mean in each arm with the corresponding trial regression. In ACTG and JTPA, we use a separate design sample to check outcome relevance and whether the arm-specific regressions predict across sources. These checks inform the choice of candidate. Consistency still requires the valid-selection assumption stated above.

AuthorsOpenReview ↗

W3 / Q2. Figure 1(c) and shadow exclusion in the simulation

Thank you for reading Figure 1 so carefully and for pointing out this path in panel (c). In the introduction, we draw an analogy with missingness not at random (MNAR). In the MNAR example, Y→RY\to R means that the outcome can affect whether it is observed. Replacing the observation indicator RR with the trial enrollment indicator SS would give Y→SY\to S. In our setting, enrollment takes place before the follow-up outcome, so we drew the selection mechanism in terms of factors present at enrollment. The arrow X→SX\to S represents selection on measured factors. The path Y(a)←U→SY(a)\leftarrow U\to S shows how an unmeasured factor can leave enrollment associated with the potential outcome even after conditioning on XX.

We intended Figure 1 as a schematic. Panel (a) shows the ignorable selection setting common in prior work. Panel (b) illustrates the nonignorable selection we allow, and panel (c) shows why a shadow variable can help in that setting. The link between ZZ and YY was meant to show that, within each trial arm, pretreatment ZZ carries information about the outcome beyond what XX tells us. To make the panels easy to compare, we used the actual treatment AA and observed outcome YY in all three. The original caption pointed readers to Section 3.2, where we state the outcome relevance and selection exclusion conditions for each potential outcome Y(a)Y(a).

But as you point out, the arrow Z→YZ\to Y naturally suggests a direct causal effect. Under that reading, even with treatment fixed at aa, conditioning on Y(a)Y(a) opens the path Z→Y(a)←U→SZ\to Y(a)\leftarrow U\to S. We are sorry the original figure made our meaning unclear. We have replaced the arrow between ZZ and YY in Figure 1(c) with a dotted line without an arrowhead to show the outcome association we intended.

Your reading of Equation (19) is also correct. In the simulation, ZZ is a noisy measurement of UU, and UU enters Y(0)Y(0) and Y(1)Y(1) with coefficients of 11 and 1.51.5, respectively. Thus, given XX, ZZ is informative about the outcome in both arms. As you observed, knowing XX and Y(a)Y(a) in either arm determines UU. The only remaining variation in ZZ comes from measurement noise independent of enrollment, so Z⊥S∣X,Y(a)Z\perp S\mid X,Y(a). We have added this explanation after Equation (19).

AuthorsOpenReview ↗

W4 / Q3. An example of a shadow variable in trial and observational data

We are sorry we did not explain clearly how a shadow variable can help us combine trial and observational data. We now develop the job-training example from the introduction in Appendix A.1 (p. 13), where we show why a worker’s pay grade can serve as a shadow variable. Machine operators can retrain as maintenance technicians. For the workers covered by local employment records, how would average base pay next year differ if everyone retrained and moved into maintenance, compared with everyone staying in their current job?

In the randomized trial (RCT), workers first volunteer, and a lottery then determines who receives training. Even among workers with the same education and years of service, those in the higher grade may gain more from retraining and therefore be more likely to volunteer. The lottery gives a fair comparison among volunteers. By the time it takes place, however, workers who stand to gain more are already overrepresented. Applying the trial average to the local workforce could therefore overstate the benefit. This is selection bias across sources.

The observational study (OBS) draws on routine employment records. It covers the target workforce and records who retrained and what each worker subsequently earned. In routine practice, supervisors may be more likely to recommend higher-grade workers for training. Those workers would earn more even if they stayed in their original jobs. A direct comparison of trained and untrained workers would therefore attribute part of a pre-existing pay gap to training. This is treatment confounding within the OBS.

In this scenario, we use a worker’s pay grade before training as a candidate shadow variable ZZ. Both studies obtain each worker’s grade directly from payroll records. Suppose the records are accurate, with Z=1Z=1 for the higher grade and Z=0Z=0 for the lower grade.

We assume that the same contractual pay schedule applies to workers in both studies and that XX includes factors used to set pay, such as years of service. In the same job, a long-serving worker in the lower grade might earn as much as a newer worker in the higher grade. For workers in the same job with the same XX, however, the schedule sets a different rate of base pay for each grade.

This pay schedule gives both shadow conditions. First, both grades are represented in each trial arm. At the same XX, workers in the two grades earn different levels of base pay, so knowing ZZ helps predict pay in either arm. This gives outcome relevance. Second, let Y(a)Y(a) denote a worker’s base pay under retraining (a=1a=1) or staying in the original job (a=0a=0). Under the assumed pay schedule, two workers with the same XX and the same Y(a)Y(a) hold the same grade. They may still differ in their willingness to volunteer. But once XX and Y(a)Y(a) are fixed, knowing whether a worker came from the RCT or the OBS gives no further information about ZZ. This gives selection exclusion, Z⊥S∣X,Y(a)Z\perp S\mid X,Y(a), in both arms, where SS denotes the data source.

We can now return to the training effect. Among lower-grade workers with the same XX, we use the randomized trial to compare average pay under retraining with average pay under staying in the original job. We make the same comparison for higher-grade workers. The OBS records tell us directly how common each grade is among target workers with that background. We use those target proportions to weight the two grade-specific gains, then average over the distribution of XX in the OBS. This gives the average gain from training for the target workforce, E[Y(1)−Y(0)∣S=OBS]\mathbb E[Y(1)-Y(0)\mid S=\mathrm{OBS}]. In this way, we use random assignment in the RCT to address the treatment confounding that arises in the OBS because supervisors are more likely to recommend higher-grade workers for training. We then use the share of each grade in the OBS to weight its training gain and correct for selection bias, since higher-grade workers are more likely to volunteer for the trial. The advantage of our method is that it addresses both biases when estimating the average gain from training for the local workforce.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.