- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper studies when historical experiments can be reused to estimate a treatment effect in a new setting. It proposes a convex combination of historical ATEs, with weights chosen to minimize a certified error bound (as one possible approach). The bound relies on uncertainty sets for latent causal mechanisms and smoothness assumptions on the treatment-effect function. The framework either produces an estimate, refuses when uncertainty is too large, or proposes additional experiments.
Critical strengths
-
The problem of determining when experimental evidence can be safely transported is important and practically relevant.
-
The paper explicitly separates sampling uncertainty from transport bias and provides finite-sample guarantees that remain valid even when the weights are selected based on the observed prior ATEs.
Critical weaknesses
- The main guarantee requires valid uncertainty sets for latent mechanisms and known upper bounds (L,H) on the smoothness of the unknown treatment-effect function. The paper does not provide a convincing procedure to obtain these quantities. The misspecificatin of these quantities would invalidate their certificate.
- The modelling is not clear to me. Why is convex combination of prior ATEs the right candidate class of estimators?
- The real data experiments do not seem to support the papers main practical claim. In the NSW experiment, the authors do not validate the assumptions needed for their guarantees. In the microcredit experiment, they lack sufficient information to establish whether results from previous experiments can reliably be applied to a new setting.
- The theoretical guarantee for choosing additional experiments holds only under a simplified linear-Gaussian model. It is unclear whether the proposed approach would work effectively in more realistic settings.
Questions for authors
Please see the weaknesses section.
5 comments
W2: Why convex weights?
We use nonnegative weights summing to one because they combine experiments while giving a tractable bound on error relative to the target. Expanding around the weighted mechanism location cancels the first-order terms, leaving differences controlled by target distance and curvature.
Let be a study's mechanism vector, its scalar weight, and the weighted location. Lemma C.1 uses
W3: Real-data evidence
NSW checks reconstruction error within one trial; microcredit still lacks sufficiently documented design information to compute an ATLAS certificate. The revised paper's complete real-data certificate calculation uses Many Labs 2 under an explicit common-category-effect model.
Appendix M now identifies NSW as an earlier implementation using point representations, with overlapping participants across local analyses. The semi-synthetic experiment sets known effects to assess error; the mechanism assumptions for the real data still require validation. For microcredit, the computed fixed-effect and range rules both refuse under the tested criteria. ATLAS has no computed error bound because reliable design coordinates are missing. Section 6.2 distinguishes these cases.
Many Labs 2 uses 40 source sites and 17 held-out targets. We retain every possible assignment of unknown categories, assume accurate labels and three exhaustive categories, and assume that each category's treatment effect is common across sites. With zero additional source bias and model residual, at tolerance 0.15:
| Weights, using design information only | Releases | Mean full interval width | Mean absolute error against noisy target references |
|---|---|---|---|
| Optimized | 8/17 | 0.3429 | 0.08134 |
| Uniform | 0/17 | 0.8662 | 0.07276 |
Both width and error are averaged over all 17 targets. Optimization produces narrower intervals and more releases, with higher point-prediction error. The certificate radius uses source sampling uncertainty; target estimates also contain noise, so the evaluation checks agreement with noisy references. Coverage of the true causal effects is not directly observed. Allowing additional source bias of 0.025 leaves one release; bias of 0.05 or a model-residual allowance of 0.025 leaves none. These results concern the retained samples under the stated measurement, retention, and response assumptions. Appendix X and WGDN-W2a/W2b/Q2 ("A real-data certificate") give the construction and comparisons.
Fyfe gives a different practical result. Under the fixed-cohort potential-response definition and consistency with observed outcomes, source information alone allows every target effect in [-1,1]. The current single-source calculation returns that entire interval. This retrospective analysis identifies the full range allowed by the available source information. For prospective transport, that range remains [-1,1] (Appendix J; WGDN-W2a/W2b/Q2 ("The Fyfe analysis")).
W1: Mechanism sets and smoothness
For Many Labs 2, we construct sets that retain every possible assignment of missing categories, then derive smoothness bounds under an explicit common-effect assumption. We also tested failures of set calibration and misspecified bounds: underestimating the slope bound led to 70 released estimates exceeding the error tolerance in 600 simulations.
Recorded Male, Female, and Other categories remain fixed; each unknown category can be any of the three. If labels are accurate and these categories exhaustive, the set contains every possible category composition of the retained sample. We then assume that each category's treatment effect is shared across sites and lies in [-1,1]. The site effect is their proportion-weighted average. In the paper's proportion coordinates and Euclidean metric, this gives
L=8/3,H=0.
Here the scalar bounds L and H limit slope and curvature; zero curvature follows from linearity. We also account for treatment/control composition imbalance and sampling error for bounded outcomes. The analysis depends on common category effects, accurate measurement, sample retention, and response assumptions. For other interventions, whether these categories capture the mechanisms affecting treatment effects needs examination.
Appendix Xgives the construction and assumptions (WGDN-W2a/W2b/Q2("A real-data certificate")).When sets are calibrated from data, coverage must hold jointly across sources, candidate experiments, and targets. We used 199 independent complete archives, each with 11 objects, for calibration and 2,000 simulated archives for evaluation:
Thus, near-95% coverage for individual objects can coexist with much lower joint coverage. Calibration also requires exchangeable archives. Multiplying errors only in new archives by 1.5 reduced maximum-error joint coverage to 37.20%; dropping descriptors only in new archives reduced it to 13.80% (
Appendix V).We separately held 600 bridgeable simulations, experiment order, and budget fixed, changing only the smoothness bounds. Weights used design information only; tolerance was 0.30:
Underestimating slope led to earlier stopping and more excessive errors; larger bounds required more participants or more refusals. Returned intervals include those at release and those returned after refusal at budget exhaustion. For a separate set of unreachable targets, at the common full-budget endpoint, underestimating slope yielded coverage of 0/600. Setting curvature to zero caused no coverage failures at that fixed endpoint; this observation is specific to that endpoint, while a global curvature bound must cover the entire nonlinear function (
Section 6.3,Appendix W).For broader interventions, mechanism sets and smoothness bounds require scientific justification specific to the intervention. A fitted curve's derivative describes that fit; the certificate requires bounds valid across the allowed function class.
Theorem 5.2controls the probability of release together with error above tolerance. Coverage conditional on release is a separate quantity requiring its own analysis (WGDN-W1a/Q1).