acceptodds
Under review as a conference paper at ICLR 2027

When Can Experiments Be Reused? Certified Transport, Refusal, and Causal Bridging

Abstract

Past experiments can help answer new causal questions, but similar study reports do not guarantee that their results apply to a new setting. Differences in populations, control conditions, or exposure can change treatment effects even when past estimates are precise. We develop a framework for deciding when experimental results can be reused. It represents uncertainty about these differences as sets and combines this uncertainty with sampling error to bound the error of a target estimate. The bound remains valid even when studies and their weights are selected after observing their results. The framework releases an estimate only when its error bound meets a prespecified tolerance. Otherwise, it reports a range of plausible effects and considers additional experiments. We prove the bound’s validity under stated assumptions, characterize what an archive cannot identify, and establish guarantees for choosing additional experiments under a restricted linear-Gaussian model. Controlled simulations show why both mechanism uncertainty and selection must be accounted for: after adaptive selection among 128 sources, simultaneous protection achieves 95.3% coverage, compared with 0.2% for unadjusted intervals. A Many Labs 2 benchmark using meta-analytic baselines illustrates how tighter precision requirements reduce the fraction of targets eligible for release. Together, these results support explicit reuse and refusal decisions, while independent calibration of mechanism uncertainty remains necessary for applications across different interventions.

Primary area:
causal reasoning
Keywords:
causality, Partial identification, Bridge experiments

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KoBTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies when historical experiments can be reused to estimate a treatment effect in a new setting. It proposes a convex combination of historical ATEs, with weights chosen to minimize a certified error bound (as one possible approach). The bound relies on uncertainty sets for latent causal mechanisms and smoothness assumptions on the treatment-effect function. The framework either produces an estimate, refuses when uncertainty is too large, or proposes additional experiments.

Critical strengths
  1. The problem of determining when experimental evidence can be safely transported is important and practically relevant.

  2. The paper explicitly separates sampling uncertainty from transport bias and provides finite-sample guarantees that remain valid even when the weights are selected based on the observed prior ATEs.

Critical weaknesses
  1. The main guarantee requires valid uncertainty sets for latent mechanisms and known upper bounds (L,H) on the smoothness of the unknown treatment-effect function. The paper does not provide a convincing procedure to obtain these quantities. The misspecificatin of these quantities would invalidate their certificate.
  2. The modelling is not clear to me. Why is convex combination of prior ATEs the right candidate class of estimators?
  3. The real data experiments do not seem to support the papers main practical claim.  In the NSW experiment, the authors do not validate the assumptions needed for their guarantees. In the microcredit experiment, they lack sufficient information to establish whether results from previous experiments can reliably be applied to a new setting.
  4. The theoretical guarantee for choosing additional experiments holds only under a simplified linear-Gaussian model. It is unclear whether the proposed approach would work effectively in more realistic settings.
Questions for authors

Please see the weaknesses section.

5 comments
AuthorsOpenReview ↗

W1: Mechanism sets and smoothness

For Many Labs 2, we construct sets that retain every possible assignment of missing categories, then derive smoothness bounds under an explicit common-effect assumption. We also tested failures of set calibration and misspecified bounds: underestimating the slope bound led to 70 released estimates exceeding the error tolerance in 600 simulations.

Recorded Male, Female, and Other categories remain fixed; each unknown category can be any of the three. If labels are accurate and these categories exhaustive, the set contains every possible category composition of the retained sample. We then assume that each category's treatment effect is shared across sites and lies in [-1,1]. The site effect is their proportion-weighted average. In the paper's proportion coordinates and Euclidean metric, this gives

L=8/3,H=0.\boldsymbol{L}=\sqrt{8/3},\qquad \boldsymbol{H}=0.

Here the scalar bounds L\boldsymbol{L} and H\boldsymbol{H} limit slope and curvature; zero curvature follows from linearity. We also account for treatment/control composition imbalance and sampling error for bounded outcomes. The analysis depends on common category effects, accurate measurement, sample retention, and response assumptions. For other interventions, whether these categories capture the mechanisms affecting treatment effects needs examination. Appendix X gives the construction and assumptions (WGDN-W2a/W2b/Q2 ("A real-data certificate")).

When sets are calibrated from data, coverage must hold jointly across sources, candidate experiments, and targets. We used 199 independent complete archives, each with 11 objects, for calibration and 2,000 simulated archives for evaluation:

CalibrationJoint coverageAdditional result
Maximum error within each archive95.60%Calibrates simultaneous coverage directly
Separate 95% calibration per object55.85%Mean marginal coverage: 94.95%
Empirical Bonferroni100%All radii infinite at this calibration size

Thus, near-95% coverage for individual objects can coexist with much lower joint coverage. Calibration also requires exchangeable archives. Multiplying errors only in new archives by 1.5 reduced maximum-error joint coverage to 37.20%; dropping descriptors only in new archives reduced it to 13.80% (Appendix V).

We separately held 600 bridgeable simulations, experiment order, and budget fixed, changing only the smoothness bounds. Weights used design information only; tolerance was 0.30:

Bounds usedReleasesMean added participantsErrors above toleranceReturned intervals covering truth
True bounds600/600160.000/600600/600
Half the true slope bound600/600107.8470/600519/600
Twice the true slope bound491/600214.240/600600/600
Twice the true curvature bound461/600190.560/600600/600

Underestimating slope led to earlier stopping and more excessive errors; larger bounds required more participants or more refusals. Returned intervals include those at release and those returned after refusal at budget exhaustion. For a separate set of unreachable targets, at the common full-budget endpoint, underestimating slope yielded coverage of 0/600. Setting curvature to zero caused no coverage failures at that fixed endpoint; this observation is specific to that endpoint, while a global curvature bound must cover the entire nonlinear function (Section 6.3, Appendix W).

For broader interventions, mechanism sets and smoothness bounds require scientific justification specific to the intervention. A fitted curve's derivative describes that fit; the certificate requires bounds valid across the allowed function class. Theorem 5.2 controls the probability of release together with error above tolerance. Coverage conditional on release is a separate quantity requiring its own analysis (WGDN-W1a/Q1).

AuthorsOpenReview ↗

W2: Why convex weights?

We use nonnegative weights summing to one because they combine experiments while giving a tractable bound on error relative to the target. Expanding around the weighted mechanism location cancels the first-order terms, leaving differences controlled by target distance and curvature.

Let mi\boldsymbol{m_i} be a study's mechanism vector, αi\boldsymbol{\alpha_i} its scalar weight, and mˉα\boldsymbol{\bar m}_{\boldsymbol{\alpha}} the weighted location. Lemma C.1 uses

=\sum_{\boldsymbol{i}}\boldsymbol{\alpha_i}\boldsymbol{m_i}, \qquad \sum_{\boldsymbol{i}}\boldsymbol{\alpha_i} (\boldsymbol{m_i}-\boldsymbol{\bar m}_{\boldsymbol{\alpha}})=0.$$ The weighted first-order changes around this location therefore sum to zero. The error bound accounts for the location's distance to the target and for source dispersion together with curvature. Nonnegative weights keep the weighted location within the convex region where the smoothness assumptions apply; a more distant target still contributes through its distance. The sampling argument is similarly direct. When every source error is bounded simultaneously, the error of any nonnegative weighted average is bounded by the same weighted average of those bounds. This holds for all allowed weights at once, so `Theorem 5.2` permits choosing weights after seeing the results. Weights chosen from design information alone admit a tighter bound; our reply to `WGDN-W1a/Q1` reports the precision comparison. Uniform, nearest-source, and inverse-variance weights all belong to this class, allowing comparisons on the same data and budget (`nrhq-W4/Q2` ("Weighting and additional experiments")). Allowing negative weights requires absolute weights in the sampling bound and additional control if the weighted location leaves the original region. We have not completed comparisons with this broader class. In the linear Many Labs 2 model, the two radius-optimization problems for each of 17 targets give 34 convex problems. In the numerical check, the computed objective exceeded a certified lower bound on the optimum by at most 2.85e-10. This check applies to the construction in `Appendix X`; general nonlinear problems require separate verification. `Section 4`, `Lemma C.1`, and `Theorem 5.2` provide the geometric and statistical arguments above. `Theorem 5.4` characterizes worst-case error across all estimators in its specified Gaussian experiment, showing how inadequate source support leaves error that increasing source sample sizes cannot eliminate. Its comparison is confined to that experiment; comparisons of convex weights with signed or nonlinear methods in general transport problems remain open.
AuthorsOpenReview ↗

W3: Real-data evidence

NSW checks reconstruction error within one trial; microcredit still lacks sufficiently documented design information to compute an ATLAS certificate. The revised paper's complete real-data certificate calculation uses Many Labs 2 under an explicit common-category-effect model.

Appendix M now identifies NSW as an earlier implementation using point representations, with overlapping participants across local analyses. The semi-synthetic experiment sets known effects to assess error; the mechanism assumptions for the real data still require validation. For microcredit, the computed fixed-effect and range rules both refuse under the tested criteria. ATLAS has no computed error bound because reliable design coordinates are missing. Section 6.2 distinguishes these cases.

Many Labs 2 uses 40 source sites and 17 held-out targets. We retain every possible assignment of unknown categories, assume accurate labels and three exhaustive categories, and assume that each category's treatment effect is common across sites. With zero additional source bias and model residual, at tolerance 0.15:

Weights, using design information onlyReleasesMean full interval widthMean absolute error against noisy target references
Optimized8/170.34290.08134
Uniform0/170.86620.07276

Both width and error are averaged over all 17 targets. Optimization produces narrower intervals and more releases, with higher point-prediction error. The certificate radius uses source sampling uncertainty; target estimates also contain noise, so the evaluation checks agreement with noisy references. Coverage of the true causal effects is not directly observed. Allowing additional source bias of 0.025 leaves one release; bias of 0.05 or a model-residual allowance of 0.025 leaves none. These results concern the retained samples under the stated measurement, retention, and response assumptions. Appendix X and WGDN-W2a/W2b/Q2 ("A real-data certificate") give the construction and comparisons.

Fyfe gives a different practical result. Under the fixed-cohort potential-response definition and consistency with observed outcomes, source information alone allows every target effect in [-1,1]. The current single-source calculation returns that entire interval. This retrospective analysis identifies the full range allowed by the available source information. For prospective transport, that range remains [-1,1] (Appendix J; WGDN-W2a/W2b/Q2 ("The Fyfe analysis")).

Reviewer_WGDNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when results from an archive of past randomized experiments can be reused for a new setting instead of running a new trial. For example, a policymaker with seven microcredit trials wants the effect in an eighth market; the trial reports read alike, but the sites differ in take-up, existing credit, and control services. The authors ask the analyst to declare, for each study, a set of ways it could differ from the target, and they combine that uncertainty with sampling error into an error bound for a weighted average of past estimates, valid even when the weights are chosen after seeing the results. An estimate is released only when the bound meets a tolerance fixed in advance. Otherwise the method reports the range of target effects the archive allows and suggests additional "bridge" experiments. Simulations check each theorem, and a Many Labs 2 benchmark scores standard meta-analysis baselines. However, the certificate itself is never applied to a real archive.

Critical strengths

The paper treats refusal as a legitimate answer and makes the refusal informative. Standard pooling and transport methods return an estimate and an interval whatever the archive looks like. Here the method can say that the archive does not support the target, and it then reports the range of target effects the archive still allows and which new experiments would narrow it. The identification result makes this point clear: similar study descriptions, including similar learned representations, cannot by themselves certify reuse. This is timely, since LLM-based tools that mine experiment archives and propose follow-up studies are starting to appear, and the paper gives a formal account of what such tools cannot certify.

The authors report results that cut against their own method. In their simulated workflow, running a new experiment beats reuse at equal cost, and they say so plainly. They also report that refusal does not lower error on one of their benchmarks. I appreciate that the unfavorable results are reported rather than left out. Together with the sealed held-out outcomes, that makes the rest of the evidence easier to trust.

Critical weaknesses

1. The selection-safe bound, the paper's main guarantee, gives up the precision that motivates pooling studies. The error bound is correct; my concern is what it costs. To stay valid when the weights are chosen after looking at the results, its noise part is a weighted sum of each study's own noise, multiplied by a factor that grows with the number of studies. No weighting can make it more precise than the single most precise study, and every added study raises the multiplier. The paper notes this itself for equally noisy studies: the selection-safe bound grows with the number of studies, while the ordinary fixed-weight bound shrinks like one over the square root of that number. The paper's simulated workflow shows the cost (Table 11). At the stricter tolerance, the selection-safe version releases no targets, the fixed-weight version releases about two-thirds, and running a new trial releases all of them. The reuse policies that do release all use the fixed-weight bound, so selection safety plays no part in them.

The abstract supports selection protection with a different result: 95.3% versus 0.2% coverage after picking the best of many sources. That comparison uses a standard Bonferroni interval, not the paper's certificate, and the fact that selecting the maximum breaks ordinary intervals is textbook. As written, the abstract and the first listed contribution lead with the version of the bound that, in the paper's own workflow, releases far less often than the fixed-weight alternative.

2. The mechanism sets that carry the guarantee are never built for a real archive, and the real-data sections test only standard baselines. The guarantee holds only if the declared sets contain the true differences between each study and the target. The paper's own Theorem 5.1 shows that the archive's data cannot check this, since two studies with identical records can imply opposite target effects. The sets must therefore come from outside knowledge, and that is the step the paper never demonstrates. The one calibration exercise uses synthetic data with known mechanism labels, which the authors acknowledge is not a procedure demonstrated on real interventions. The Many Labs 2 holdouts score fixed- and random-effects pooling, meta-regression, and conformal intervals, which the paper describes as model-based comparators rather than applications of its certificate. The microcredit pilot, the paper's motivating example, refuses at every tolerance because the provenance of the design information could not be resolved. So the title question is never answered for a real archive, not even with deliberately wide sets that would end in refusal.

I also don't think the Many Labs 2 benchmark can stand in for that test. The framing effect is nearly identical across sites, and as far as I can tell from the radius formula the paper uses, whether a held-out site is released at the paper's threshold depends mainly on that site's own sample size. The release frontier in Figure 3 then reflects how large each site is more than whether the archive supports transport to it.

This matters because the theory, while correct, is built from standard tools: a union bound, a Taylor bound, a Lipschitz extension, and a two-point lower bound. The contribution therefore rests on the framework being usable, and the paper has not yet shown that it is.

A smaller point on presentation: the appendix tables call the method "Causal ATLAS," a name the main text never introduces, and one appendix reports an earlier prototype rather than the method the theorems cover, which makes the appendix hard to follow.

Questions for authors

Questions whose answers could change my assessment

  1. Is there any setting, in your simulations or on real data, where the selection-safe radius releases a target that the fixed-weight radius (with design-only weights, or with sample splitting) refuses? If not, what practical role do you see for Theorem 5.2?

  2. How would you construct the mechanism sets for the Many Labs 2 effect families or the seven microcredit trials without mechanism labels, and what would the certificate report on those data? An answer that ends in refusal for every target would still help me judge whether the framework can be used.

  3. The pairwise constraints behind Theorem 5.3 look like the bounded-variation parameter space that Ishihara and Kitagawa use for treatment choice from several published trials ("Evidence Aggregation for Treatment Choice," arXiv:2108.06473), building on Manski ("Toward Credible Patient-centered Meta-analysis," Epidemiology, 2020). How does Theorem 5.3 relate to that work, and what does it add?

8 comments
AuthorsOpenReview ↗

W1a/Q1: Selection and precision

W1a/Q1: Selection and precision

With matched inputs and confidence budgets, the globally optimized design-only rule releases every target that the selection-safe rule can release, provided weights can be chosen using design information alone. Theorem 5.2 covers choosing studies or weights after seeing historical outcomes; compared with honest sample splitting, each approach releases some targets that the other refuses.

The precision cost comes from the sampling-error term. For the selection-safe rule, this term is a weighted average of source noise scales, multiplied by a factor that protects all sources simultaneously. Its minimum is therefore the smallest source noise scale times that factor, which increases with the number of sources. With K\boldsymbol{K} independent, equally noisy sources, the sampling term for uniform design-only weights shrinks as 1/K1/\sqrt{\boldsymbol{K}}, while the selection-safe term grows as log⁡K\sqrt{\log\boldsymbol{K}}. Adding studies can still improve the match to the target and thereby reduce transport error.

With common noise scale s\boldsymbol{s} and failure budget ζ\boldsymbol{\zeta}, the sampling terms for uniform design-only weights and selection-safe weights are, respectively,

\qquad \boldsymbol{s}\sqrt{2\log(2\boldsymbol{K}/\boldsymbol{\zeta})}.$$ Thus, if the selection-safe rule meets the release threshold, the optimized design-only rule also meets it with the same inputs. Writing $\boldsymbol{R}_2$ and $\boldsymbol{R}_{\infty}$ for the full radii, for any nonnegative weights $\boldsymbol{\alpha}$ summing to one, $$\min_{\boldsymbol{a}\in\boldsymbol{\Delta}_{\boldsymbol{C}}} \boldsymbol{R}_2(\boldsymbol{a};\boldsymbol{\zeta}) \le \boldsymbol{R}_{\infty}(\boldsymbol{\alpha};\boldsymbol{\zeta}),$$ where $\boldsymbol{\Delta}_{\boldsymbol{C}}$ is the weight simplex over the sources. This compares the sizes of the bounds. The design-only guarantee requires choosing weights without using the outcome noise on which they are evaluated; applying a fixed-weight guarantee after outcome-based selection can lose coverage. `Section 5` and `Appendix C` now explain these two conditions separately. `Appendix U` compares the certificates across 360,000 independent simulated worlds, with known Gaussian noise, sets containing the true mechanisms, and valid smoothness bounds. The table shows the prespecified setting with 16 equally precise sources, zero effects, and strong selection. Methods use the same source resources, nominal 95% coverage, and release tolerance 0.30. | Method | Empirical coverage | Mean full interval width | Release rate | |---|---:|---:|---:| | Outcome-based weights, selection-safe bound | 99.40% | 0.7190 | 0% | | Optimized design-only weights | 99.44% | 0.1358 | 100% | | Honest half-sample split | 99.50% | 0.7683 | 0% | | Outcome-based weights, unadjusted fixed-weight bound | 89.56% | 0.5432 | 100% | Across the prespecified configurations, selection strengths, and thresholds, there were 48,176 decisions where the selection-safe method released and splitting refused, and 331,870 in the reverse direction. These counts include multiple decisions on the same simulated world. The 15,000 release reversals relative to design inverse-variance weighting occurred in particular weak-selection settings with unequal source precision; those comparisons also vary weights, source and target locations, and selection. In the original additional-experiment workflow, at tolerance 0.20, the target-focused selection-safe rule released 0/600 bridgeable targets, the design-only rule 399/600, and direct target trials 600/600. Successful historical reuse here used the tighter design-only bound (`Section 6.3` and `Appendix P`). Participant costs are compared in our reply to `nrhq-W4/Q2`, “Weighting and additional experiments.” We therefore use the design-only rule when weights can be fixed without seeing outcomes, and `Theorem 5.2` when selection uses outcomes. The theorem controls the probability of jointly releasing an estimate and exceeding the error tolerance. Coverage calculated only among released targets is a separate conditional quantity.
AuthorsOpenReview ↗

W1b/W3: Attribution and implementation

The 95.3% versus 0.2% comparison uses Gaussian Bonferroni intervals, and we have corrected the abstract to name that method. We also introduce Causal ATLAS in the Introduction and identify the NSW implementation in Appendix M as an earlier prototype that represents each study by a single point.

In that comparison, selection among 128 sources gives coverage 0.9526 for Bonferroni intervals and 0.00158 for unadjusted intervals. This illustrates the need to account for outcome-based selection. A separate matched comparison in Appendix U evaluates the full certificate's coverage, width, and release rate; see our reply to WGDN-W1a/Q1, “Selection and precision.”

Appendix M and Section 6.2 now distinguish NSW reconstruction within one trial, ordinary prediction comparisons on Many Labs 2, and certificates computed under explicit model assumptions. Each set of results is thus tied to the method used and the question it addresses. The NSW analysis examines within-trial reconstruction; we discuss the validation status of its mechanism assumptions in our reply to KoBT-W3.

AuthorsOpenReview ↗

W2a/W2b/Q2: A real-data certificate

We have computed certificates using 40 source sites and 17 held-out targets from Many Labs 2. Assuming that category-specific treatment effects are shared across sites, optimized design-only weights release 8/17 targets at tolerance 0.15; allowing additional bias or model residuals substantially reduces releases.

We start with the recorded Male, Female, and Other counts. Participants with missing categories remain in the sample and may belong to any of the three categories. The resulting sets contain every category proportion consistent with those records. They describe the retained participants and require accurate recorded labels and categories that cover everyone. If l\boldsymbol{l} contains the recorded category counts divided by sample size and r\boldsymbol{r} is the fraction with unknown membership, the set is

\qquad \boldsymbol{\Delta}_3= \\{\boldsymbol{p}\in\mathbb{R}_+^3: \mathbf{1}^{\top}\boldsymbol{p}=1\\}.$$ Cross-site reuse additionally assumes that each category has the same treatment effect across sites, within [-1,1]. A site's effect is then the category effects weighted by its category proportions. Writing $\boldsymbol{\gamma}$ for the shared category effects, the model and its smoothness bounds, using the paper's affine extension from the Euclidean simplex, are $$\boldsymbol{\mu}(\boldsymbol{p}) =\boldsymbol{p}^{\top}\boldsymbol{\gamma}, \quad \boldsymbol{\gamma}\in[-1,1]^3, \qquad \boldsymbol{L}=\sqrt{8/3}, \quad \boldsymbol{H}=0.$$ Differences in category proportions between source treatment and control arms enter as a bias term; the sampling bound for bounded outcomes uses conditional independence of responses. The conclusions concern the retained cohort under the stated treatment, measurement, selection, and reporting conditions. `Section 6.2` and `Appendix X` describe this construction and its assumptions. The table uses the same source data, zero additional source bias and model residual, and an illustrative tolerance of 0.15. Width and mean absolute error (MAE) are averaged over all 17 targets. MAE is measured against held-out risk differences that contain sampling noise. | Weighting rule | Error bound | Mean full width | Releases | Reference MAE | |---|---|---:|---:|---:| | Uniform | Design-only | 0.8662 | 0/17 | 0.07276 | | Design inverse-variance | Design-only | 0.7964 | 0/17 | 0.07389 | | Optimized design-only | Design-only | 0.3429 | 8/17 | 0.08134 | | Optimized outcome-adaptive | Selection-safe | 0.6178 | 0/17 | 0.09342 | | Fixed-effect | Selection-safe | 1.4012 | 0/17 | 0.07409 | | DL random-effects | Selection-safe | 1.4211 | 0/17 | 0.07382 | The fixed-effect and DerSimonian–Laird (DL) weights here use observed outcomes, so we apply the bound that permits outcome-adaptive selection. `Appendix X` separately reports the intervals originally used by these methods, which differ from these certificates in the quantities they estimate and the assumptions they use. Optimizing weights produces narrower certificates and more releases, with higher MAE than uniform weighting. The optimized intervals contain 16/17 raw reference values; all 17 become compatible after accounting for reference uncertainty. This evaluates agreement with noisy references; coverage of the underlying causal effects is not directly observed. Holding weights fixed, an extra source-bias allowance of 0.025 reduces releases from 8 to 1. A bias allowance of 0.05, a source–target residual allowance of 0.025, or doubling the noise variance each reduces releases to 0. Our reply to `KoBT-W1` reports further consequences of misspecified inputs. This construction applies to these records under the shared category-effect model. For other effect families, complete sets of possible underlying mechanisms remain to be established. The seven microcredit trials also lack design coordinates with reliable provenance, so ATLAS certificates have not been computed for them. Our reply to `WGDN-W2a/W2b/Q2`, “The Fyfe analysis,” gives a completed calculation that retains the full target range.
Reviewer_nrhqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper claims to develop a framework for deciding when the results of one or more experiments can be reused to infer the causal effects under alternative conditions that would normally require an additional experiment.

Critical strengths

The topic of the paper is an interesting one, though there exists a vast relevant literature.

Critical weaknesses

The writing in the paper is overly technical and difficult to follow. For example, the third paragraph reads: “Figure 1 links the archive–target mismatch to our certificate and its transport, refusal, and bridge-design decisions.” Several terms in this sentence are undefined and unlikely to be know by the average reader. Similarly, a sentence in the fourth paragraph reads: “The decision layer remains open when compatibility is uncertain and studies or weights may be selected after outcomes are observed.” It is unclear (to this reader at least) what a “decision layer” refers to, and the term is never defined and only used one other time in the paper. Another example is the caption for Figure 1, which reads, in part: “Audited mechanism sets and simultaneous noise control yield a selection-safe target-error radius after outcome-adaptive weighting.”

In general, terms are rarely defined or introduced before they are used, intuitions and examples about the technical results are rarely provided, and nearly non-sensical or vacuous statements appear to be common throughout the narrative. One likely reason is that generative AI tools have been used to produce nearly everything about the paper, as indicated by the AI Use Statement. I did not see this statement until I got to the end of the paper, though I had already inferred this based on the quality of writing and explanation.

The related work section is one paragraph, despite the fact that there exists a vast literature on external validity, transportability, meta-analysis, and other relevant topics. Appendix A is much larger and more detailed, but the main paper should do a far better job of explaining where the authors’ work fits within the larger literature and how it advances the frontier.

The authors do not compare their proposed approach to viable alternatives.

Questions for authors

What are the key ideas here? In simple terms, what innovative ideas allows your method to do a better job of experiment selection, experiment weighting, and decisions regarding release of an estimate?

How do your results compare with alternative approaches?

6 comments
AuthorsOpenReview ↗

W1: Clear definitions

We have rewritten the Introduction passages and Figure 1 caption you highlighted, explaining the decision before introducing the terminology. The opening now states which new setting we want to estimate an effect for, how much support the existing experiments provide, and when the error is too large to proceed with reuse.

By a "mechanism," we mean conditions that can change an experiment's effect, such as the extent of exposure to the intervention or the services received by the control group. When these conditions are incompletely known, we retain a set of possibilities, called a "mechanism set." We then combine differences between settings, bias in the source estimates and sampling error into a bound on the target estimate's error. The paper calls this bound a "certificate"; it depends on the assumptions stated in the paper.

Selecting studies means deciding which receive weight. The weighting considers both their precision and their differences from the target. We release an estimate when its error bound meets a prespecified tolerance. Otherwise, we report the effects still allowed by the evidence, or an interval containing them.

These changes appear in the Introduction and Figure 1. Section 3.1 then defines mechanisms, study records and mechanism sets, and Section 4 explains the components of the bound. Appendix B provides a worked numerical example, discussed in our reply to nrhq-W2/Q1, "Main idea."

AuthorsOpenReview ↗

W2/Q1: Main idea

Our idea is to bound the error from using historical results for the new target, then use that bound to select studies, choose weights and decide whether to release an estimate. A historical study can be very precise while its differences from the target still matter for that decision.

For example, Appendix B combines a bound of 0.06 for differences between settings and source-estimation bias with a sampling bound of 0.04. The resulting radius is 0.10 and the full interval width is 0.20. If the prespecified error tolerance is 0.15, the estimate can be released. Keeping the source estimates unchanged but raising the first component to 0.14 gives a radius of 0.18, so release is refused. What we know about the differences between settings therefore affects whether reuse is supported.

The weighting follows the same reasoning: it considers source-target distance, how the effect function bends as conditions change, and the precision of the source estimates. Theorem 5.1 explains why this information is needed. Two underlying worlds can produce the same observed archive while having different target effects. Increasing the precision of historical estimates can leave that ambiguity unresolved.

For weights chosen after examining outcomes, Theorem 5.2 places all allowed weights under a common error guarantee. The cost can be wider intervals; weights based only on design information admit a tighter bound. Our reply to WGDN-W1a/Q1 compares this precision cost, and our reply to KoBT-W2 explains the choice of convex weights.

If the bound remains too large, Theorem 5.3 calculates which target effects are still possible. It applies to a model with known mechanism locations and effect differences bounded by distance. Additional experiments may narrow this range, and two experiments may need to be conducted together to help. The greedy experiment-selection guarantee is restricted to the stated nonadaptive linear-Gaussian model, as discussed in our reply to KoBT-W4.

Section 5 now introduces the theory through these questions, and Appendix B gives the example above. The AI Use Statement retains the disclosed assistance and describes code tests, arithmetic checks and author review against recorded outputs; the authors are responsible for the claims, proofs and explanation. Practical comparisons appear in our two replies to nrhq-W4/Q2, "Weighting and additional experiments" and "Real-data results."

AuthorsOpenReview ↗

W3: Relation to prior work

We have expanded Section 2 from one paragraph to five, explaining the problems addressed by related work, the methods we build on and what we add. The paper focuses on a specific decision: whether existing experiments support an estimate for a new setting at a given error tolerance.

External-validity and trial-generalization methods connect populations through mechanism assumptions, covariates, participation assumptions and overlap; hierarchical meta-analysis models differences in effects across studies. We use mechanism sets to retain unresolved differences between the source and target settings, then combine them with bounds on effect variation to bound the weighted estimate's target error. These sets and variation bounds require scientific justification.

Theorem 5.3 is most closely related to partial identification. Manski intersects the target-effect ranges allowed by individual studies; Ishihara and Kitagawa restrict differences between study effects to guide treatment choice. We use such constraints to find the smallest and largest target effects at a finite set of known mechanism locations, and use established Lipschitz extension results to show that the endpoints are attainable. Here, a Lipschitz constraint bounds an effect difference by a specified multiple of the distance between conditions. The resulting range describes what the evidence still allows after point release is refused. Our reply to WGDN-W2d/Q3 gives the detailed correspondence.

Section 2 also explicitly credits ExAtlas with experiment composition, local-support assessment and suggestions for additional experiments. Our formulation further handles unresolved mechanism locations, includes source-estimation bias and provides simultaneous error control for weights chosen after observing outcomes. The existing comparison concerns only the composition component of the legacy point-representation implementation; a matched comparison of the two complete current systems remains outstanding.

Simultaneous inference and log-determinant experimental design are also established tools used in the paper. Our theoretical analysis gives their specific guarantees in this reuse problem and explains experiment complementarity and the conditions for the additional-experiment guarantee. Section 2 and Appendix A have been revised accordingly; our reply to KoBT-W4 discusses the bridge-design conditions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair

We have clarified how to use the framework and added computations on real archives. Under the stated models, optimizing weights narrows error intervals and supports more releases; allowing larger bias reduces those releases, and some archives still leave the entire target range possible.

1. How to use the framework

The Introduction and Figure 1 now explain the target effect, which differences from historical experiments may change it, and how those differences combine with sampling error. Appendix B then shows, with numbers, why the same historical estimates can lead to different release decisions under different transport assumptions. The terminology is introduced through these steps (nrhq-W1 and nrhq-W2/Q1; WGDN-W3).

For Many Labs 2, we retain all possible completions of unknown recorded category membership, then derive the error bound under a common category-effect assumption across sites. This provides a complete path from records to calculation. Applying the approach to more complex interventions requires corresponding scientific support; the sensitivity analysis shows what happens when the supplied assumptions are wrong (WGDN-W2a/W2b/Q2, "A real-data certificate"; KoBT-W1).

2. What the theory adds

The framework combines uncertainty about experimental differences, bounds on effect variation, source-estimation bias and sampling error in one target-error bound, then uses that bound to decide whether to release an estimate. Error control that holds for all allowed weights lets analysts choose studies and weights after inspecting historical outcomes. This flexibility has a precision cost. The comparison across 360,000 simulated worlds measures that cost and identifies cases where this approach and sample splitting release different targets (nrhq-W2/Q1; WGDN-W1a/Q1; KoBT-W2).

Section 2 explains the connection to established tools, including simultaneous inference, Lipschitz extension and log-determinant design. Our reply to WGDN-W2d/Q3 compares Theorem 5.3 with Manski and Ishihara-Kitagawa; the reply to KoBT-W4 explains why experiments may need to be combined and which model supports the greedy guarantee. The abstract now attributes the 95.3% versus 0.2% example to Gaussian Bonferroni protection (nrhq-W3; WGDN-W1b/W3).

3. What the comparisons show

Many Labs 2 uses 40 source sites and 17 targets. Under the common category-effect model and the stated conditions, optimized design-only weights release 8/17 targets at tolerance 0.15, compared with 0/17 for uniform weights. Across all 17 targets, mean width falls from 0.8662 to 0.3429, while mean absolute error against noisy references rises from 0.07276 to 0.08134. Allowing extra source bias of 0.05 removes all releases. We also corrected the interpretation of the original predictive benchmark, where release largely follows target sample size (nrhq-W4/Q2, "Real-data results"; WGDN-W2a/W2b/Q2 ("A real-data certificate") and WGDN-W2c; KoBT-W3).

The completed Fyfe sensitivity analysis holds the records fixed, and its retrospective one-source calculation returns the target interval [-1,1]. NSW provides a within-trial reconstruction check; microcredit lacks the design information needed to compute a certificate. Synthetic comparisons hold data and budgets fixed: targeted bridges use fewer participants than random or nearest bridges, while a direct target trial is cheaper in this setting. The comparisons and their conditions are detailed in nrhq-W4/Q2 ("Weighting and additional experiments"), WGDN-W2a/W2b/Q2 ("The Fyfe analysis"), KoBT-W3, and KoBT-W4.

Thank you again for the time and care invested in these reviews. We welcome further questions and would be happy to discuss any aspect of the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.