acceptodds
Withdrawn from ICLR 2027

Beyond Exchangeability: Distribution-Shift-Aware Integration of External Control Data in Randomized Trials

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Randomized controlled trials (RCTs) are the gold standard for evaluating causal effects but are often costly and difficult to scale; consequently, they are frequently augmented with auxiliary external controls in many applications. Prior approaches to borrowing such data typically rely on exchangeability, under which the external controls are readily usable for inference in the trial population. In practice, however, differences in eligibility criteria, standard of care, and data collection procedures may induce distribution shifts between the RCT and the external controls, making exchangeability implausible. In this paper, we propose a general framework for incorporating external controls that explicitly accounts for distribution shifts through an augmented estimating function, allowing information about the shifts to be leveraged for efficiency gains even when exchangeability does not hold. To guard against potential bias and efficiency loss due to misspecification of the distribution shift, we further develop an adaptive shrinkage estimator that preserves consistency while guaranteeing efficiency dominance over the trial-only benchmark. Synthetic experiments and a real data application demonstrate the practical advantages of the proposed approaches.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Adaptive shrinkage, causal inference, distribution shift, external control data, exchangeability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tXxGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how external control data can improve estimation precision in randomized trials when the two data sources have different distributions. The method adjusts for differences in covariates and conditional outcomes. It also uses adaptive shrinkage to rely more on trial data when the models are misspecified, under certain conditions. The paper provides consistency and asymptotic efficiency analyses, supported by simulations and a real-data application.

Critical strengths
  1. Trial participants and external controls often differ in practice. The method directly accounts for these differences, which gives the work a clear motivation.

  2. The method combines distribution calibration with adaptive shrinkage. It also uses covariate information from external controls to estimate the mean outcome in both trial arms more precisely. This is a useful contribution.

  3. The theory covers consistency, efficiency gains, and uncertainty from estimating distribution shifts. The authors also explain the limits of the method under local model misspecification.

Critical weaknesses
  1. The gains in precision rely on reliable information about how the trial and external populations differ. Although several possible sources are discussed, I am not sure how confidently these differences can be modeled in practice, especially for outcomes. It is also unclear when these assumptions are more reasonable than exchangeability.

  2. It is unclear to me when the method offers enough practical benefit to justify the extra modeling and validation data. Shrinkage can protect the estimate by falling back on trial data, but this may also leave little gain from the external controls. The settings where the method is most useful remain unclear.

  3. The real-data example includes six covariates, but the simulations use only one. This leaves me unsure how the method performs when several covariates differ between the two populations. The real-data results alone cannot establish estimation accuracy because the true treatment effect is unknown.

Questions for authors
  1. What makes the assumptions about population differences credible in a real application? When would they be easier to justify than exchangeability?

  2. In which practical settings are the gains likely to justify the extra modeling and validation data? When would using trial data alone be preferable?

  3. How does the method perform when several covariates differ between the two populations? Do the gains in precision and the confidence interval coverage remain similar?

Reviewer_q4NEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers a basic difficulty in using external controls: once the trial and external populations have different outcome distributions, borrowing more data can introduce bias rather than improve precision. The authors address this by explicitly modeling the covariate and outcome distribution shifts. The resulting calibration equations allow external observations to contribute to estimation of both arm-specific means. The main theoretical results describe the variance reduction with known shifts and the additional uncertainty with estimated shifts. An adaptive combination with a trial-only estimator provides a fallback when the augmented estimator has a fixed asymptotic bias. Simulations and an NSW–PSID analysis illustrate these results.

Critical strengths

The strongest part is the treatment of shift estimation. Proposition 1 alone would give a rather optimistic picture based on known weights. Theorem 2 makes clear that estimating these weights adds first-order uncertainty and can remove the apparent gain. This is a useful qualification, and the paper does not hide it. I also find the discussion in Appendix A valuable. It explicitly derives the shifted limiting distribution under local misspecification and explains why the adaptive rule cannot reliably distinguish such bias from sampling noise. This helps identify the actual scope of the shrinkage result.

Critical weaknesses
  1. The source of the empirical efficiency gain is not sufficiently isolated. The real-data analysis constructs a validation sample by splitting the available data. These held-out observations include randomized trial data. Such observations are useful not only for learning distribution shifts, but also for estimating the treatment effect directly. The relevant comparison is therefore with a trial-only procedure using all available randomized observations. It is not clear whether Table A.6 makes this comparison.

This matters for the interpretation of Experiment II as well. A larger validation sample supplies more information about the shift, but may also supply more trial information. The experiment demonstrates that additional information helps; the paper needs to show how much of the gain is specifically attributable to the proposed use of external controls. The known-θ experiment answers a different, more idealized question.

  1. The shrinkage guarantee leaves an important practical regime insufficiently examined. Theorem 3 separates zero asymptotic bias from a fixed nonzero bias. In a small trial, however, an external estimator can have a consequential bias that is still difficult to distinguish from sampling error. Appendix A acknowledges this issue, but the numerical results do not examine it systematically.

I do not regard the local-misspecification limitation as a contradiction of the theorem. My concern is that “guaranteed consistency and efficiency dominance” can be read more broadly than the pointwise result warrants. A variance reduction does not by itself imply lower MSE when borrowing retains bias. The coverage of 0.914 for the treatment mean in scenario S3 of Table A.5 also suggests that this distinction has finite-sample consequences. The proposed bias-aware alternative deserves numerical evaluation if it is intended to address this concern.

Questions for authors
  1. In Table A.6, does the trial-only estimator use the randomized observations from both halves of the split? If not, please provide that comparison. Please also specify the composition of the validation sample in Experiment II.

  2. Which shrinkage construction is used for the ATE? In Table A.6 without validation, the reported arm-specific estimates give (2.444-1.787=0.657), whereas the ATE is 0.907. This could be explained by applying shrinkage directly to the ATE, as discussed in Remark 1. Please state whether this is the case and distinguish its guarantee from that of separate arm-specific shrinkage.

  3. How does the method behave as the shift-model error increases gradually from zero? I would like to see MSE and coverage, together with the borrowing weights, rather than only several fixed misspecification cases. The sensitivity to γ and the performance of the Appendix A intervals are particularly relevant here.

  4. Does the covariance estimator used in the experiments include the uncertainty from estimating shifts on the validation sample? Please give enough detail to reproduce the shrinkage weights and confidence intervals.

Reviewer_hKUsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a framework for integrating external controls into RCTs without assuming exchangeability. By modeling distribution shifts, the authors develop augmented estimators to improve efficiency. An adaptive shrinkage estimator is further proposed to maintain consistency under model misspecification.

Critical strengths

The paper addresses an important challenge in external control borrowing, namely, the non-exchangeability of ECs. The proposed framework is interesting because it explicitly leverages information about non-exchangeability to improve efficiency while preserving consistency through adaptive shrinkage.

Critical weaknesses
  1. The role of validation data raises main concerns. The validity of the validation data still relies on assumptions about their representativeness. This may simply shift the exchangeability concern from external controls to the validation data. If such high-quality validation data are available, why not directly incorporate them into treatment effect estimation? It is difficult to imagine realistic settings where RCT, EC, and additional validation data are all available. In the LaLonde application, the authors artificially split the data into estimation and validation sets. In practice, using all available data for estimation seems more reasonable. Overall, I find the emphasis on validation data somewhat misleading in the context of integrating external controls into RCTs.

  2. The magnitude and source of efficiency gains need more discussion. As shown in the first simulation experiment, without validation data, meaningful efficiency gains seem to depend on accurate knowledge of the concept shift. Without such knowledge, I doubt whether substantial efficiency gains are achievable. The authors should clarify this limitation rather than overemphasizing the advantage of not requiring exchangeability, since this comes at the cost of accurately modeling distribution shifts.

Questions for authors
  1. Some claimed advantages, such as not requiring exchangeability and leveraging external information for the treatment arm, also hold for prognostic adjustment with efficient estimators (Liao et al., 2025). Could the authors provide a more detailed comparison, particularly in the numerical experiments? Prognostic adjustment with efficient estimators is asymptotically equivalent to an efficient trial-only estimator, so its benefits are mainly seen in small RCTs. Does the same hold for the proposed method? The current simulations use relatively large sample sizes, so it would be helpful to evaluate performance with smaller RCTs.

  2. What is the percentage of efficiency improvement achieved by the proposed method in the numerical experiments? Since trial-only estimators already provide valid inference, demonstrating consistency alone is not sufficient. The magnitude of efficiency gains is important for assessing the practical value of the proposed method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.