acceptodds
Under review as a conference paper at ICLR 2027

Designing Experimental Selection Correction with Multiple Secondary Outcomes

Abstract

Researchers increasingly have access to observational data containing primary and multiple secondary outcomes, together with randomized experiments in which only secondary outcomes are measured. Experimental selection correction (ESC) leverages experimental evidence to correct selection bias in observational estimation of the primary average treatment effect (ATE). Existing ESC methods, however, typically treat the set of measured secondary outcomes and the treatment-assignment rule as fixed, resulting in potential efficiency loss. We address this limitation by developing a novel experimental design framework that jointly optimizes secondary-outcome selection and treatment assignment under resource constraints, accounting for the tradeoff between the informativeness and measurement costs of secondary outcomes. We characterize the oracle design that minimizes the semiparametric efficiency bound for estimation of the ATE. To learn the design, we propose EfficientESC, a two-stage procedure that uses observational data and a randomized pilot to determine the design for the main experiment. We show that the learned design converges to the oracle design, and the proposed ATE estimator is asymptotically normal and attains the minimized semiparametric efficiency bound. Extensive simulations demonstrate substantial efficiency gains over competing methods in finite samples.

Primary area:
causal reasoning
Keywords:
Active data collection, Causal inference, Data fusion, Experimental design, Semiparametric efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uPhMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EfficientESC, a two-stage experimental design framework that jointly optimizes secondary outcome selection and treatment allocation under budget constraints. The authors characterize an oracle design that minimizes the semiparametric efficiency bound for average treatment effect (ATE) estimation, and establish theoretical guarantees for design recovery and valid inference. Simulation results show that the proposed method achieves efficiency improvements relative to several competing methods.

Critical strengths
  1. The paper addresses an important experimental design problem by jointly optimizing secondary outcome selection and treatment allocation under budget constraints.

  2. The proposed framework explicitly accounts for the trade-off between measurement cost and statistical efficiency, providing a principled approach to experimental design.

  3. The theoretical analysis establishes oracle design recovery, asymptotic normality, and semiparametric efficiency under the stated assumptions.

Critical weaknesses
  1. The framework assumes that all candidate secondary outcome sets satisfy latent unconfoundedness, which is difficult to verify in practice and may significantly limit its applicability.

  2. The experimental evaluation relies entirely on synthetic data and includes only three nested secondary outcome sets, so its generalizability to more complex real-world settings has not been fully established.

  3. The proposed method excludes pilot observations from final ATE estimation, potentially sacrificing finite-sample efficiency. Appendix D.3.5 also shows that some full-budget baseline designs outperform EfficientESC at smaller sample sizes.

  4. The framework assumes that measurement costs are known and considers an expected budget constraint, which limits its practical applicability when costs are uncertain or when a strict budget limit must be satisfied.

Questions for authors
  1. How robust is EfficientESC when some candidate secondary outcome sets violate the latent unconfoundedness assumption?

  2. Can the authors provide additional experiments involving more secondary outcomes, complex covariates, or real-world datasets to demonstrate the generalizability of the method?

  3. Would incorporating pilot observations into the final ATE estimation improve finite-sample efficiency, especially when the experimental budget is limited?

  4. How can the proposed framework be extended to handle uncertain measurement costs and strict finite-sample budget constraints?

7 comments
AuthorsOpenReview ↗

Response to Reviewer uPhM (Part 1/5)

Thank you for your constructive comments. We apologize for the delayed response, as we devoted considerable time to extensive revisions prompted by your feedback. We would be grateful for your review of our revised manuscript and responses.

W3-1. “The framework assumes that all candidate secondary outcome sets satisfy latent unconfoundedness, which is difficult to verify in practice and may significantly limit its applicability.”

Response. Your concern is about how candidate-set validity enters the design problem. We clarify the substantive identification requirement, explain the roles of the baseline outcomes and add a simulation in which the learner is also offered invalid candidates.

(1) In response to your concern about the identification requirement: Latent Unconfoundedness (LU) is an identifying assumption inherited from ESC [1]. As clarified in Section 3.1 (page 3), the candidate collection is prespecified based on substantive knowledge and is assumed to contain only secondary-outcome sets satisfying the identification conditions. Our framework takes the validity of these candidates as an input and focuses on selecting among them to optimize statistical efficiency, rather than testing or establishing their validity from the data. A related distinction arises in instrumental-variable research: substantive arguments motivate the validity of candidate instruments, while statistical procedures can select among candidates but do not, by themselves, establish their validity. Angrist and Krueger [2] motivate quarter of birth as an instrument through compulsory schooling laws, Card [3] use college proximity as an instrument, and Belloni et al. [4] develop statistical methods for selecting instruments. To examine the consequences of violating the prespecification requirement, Appendix D.3.5 (pages 42–45) additionally includes LU-invalid candidate sets and evaluates design recovery, MSE, and confidence-interval coverage (see Q3-1 below).

(2) In response to your concern about the roles of the simulated secondary outcomes: In our DGP, S1S_1 supplies the identifying information, S2S_2 adds prognostic information, and S3S_3 is redundant conditional on S2S_2. Appendix D.3.5 (pages 42–45) additionally examines invalid candidates, with results reported in Q3-1 below.

AuthorsOpenReview ↗

Response to Reviewer uPhM (Part 2/5)

W3-2. “The experimental evaluation relies entirely on synthetic data and includes only three nested secondary outcome sets, so its generalizability to more complex real-world settings has not been fully established.”

Response. Your concern asks for evaluation beyond three nested outcome sets and a binary covariate, as well as a clearer account of the real-data motivation. The revision adds a fourth outcome with non-nested candidates and a continuous-covariate setting.

(1) In response to your concern about the motivating application and required data sources: Section 6 (page 9) explains that the NYC administrative records required by the motivating STAR/NYC application are not publicly available. Moreover, applying ESC requires observational and experimental data sources with corresponding secondary outcomes satisfying the identification conditions in Assumption 1. The current study therefore focuses on simulation-based methodological evaluation. We acknowledge the importance of empirical validation using suitable datasets and identify it as a direction for future work in Section 6.

(2) In response to your concern about the role of synthetic data: Simulation allows us to compare alternative treatment-assignment rules under controlled data-generating processes. For each simulated unit ii, we specify both potential secondary outcomes, Si(1)S_i(1) and Si(0)S_i(0). In observed data, only the potential outcome corresponding to the treatment actually received is observed for a given individual [5]. Specifying both potential outcomes in the simulation therefore allows us to evaluate alternative assignment rules counterfactually on the same underlying population and compare their performance with known treatment effects and oracle design criteria. As illustrated by the methodological studies discussed in W2-1(6), simulation-based evaluation is commonly used in adaptive experimental design. Related evaluations also include simulations calibrated to experimental data [6, Section 5] and synthetic case studies based on real-world data [7].

(3) In response to your concern about more outcomes and continuous covariates: Our additions directly test the design-learning procedure with a larger, non-nested candidate collection and a continuous covariate. Appendix D.3.8 (pages 46–48) adds S4S_4 and compares all eight sets in A4=1∪B:B⊆2,3,4\mathcal A_4=\\{\\{1\\}\cup B:B\subseteq\\{2,3,4\\}\\}, including non-nested candidates. Appendix D.3.9 (pages 48–49) uses X∼Unif[0,1]X\sim\mathrm{Unif}[0,1], covariate-dependent outcome coefficients and the prespecified class

RA=ϱA:ϱA(x)=(1−x)pA,0+xpA,1,(pA,0,pA,1)∈[0.1,0.9]2.\mathcal R_A=\\{\varrho_A:\varrho_A(x)=(1-x)p_{A,0}+xp_{A,1},\quad (p_{A,0},p_{A,1})\in[0.1,0.9]^2\\}.

Binary response nuisances use logistic regression, and continuous response nuisances use natural cubic splines with four basis functions. Figures D.13–D.14 (page 48) and D.15–D.16 (page 49) report variance and recovery. The table below adds coverage at n=5000n=5000.

SettingRecovery (%)Normalized variance (τ^\widehat\tau)Normalized variance (τ^+\widehat\tau_+)Coverage (%) (τ^\widehat\tau)Coverage (%) (τ^+\widehat\tau_+)
Four outcomes96.7019.63017.87194.794.5
Continuous XX97.4617.69916.08594.894.8

The added experiments show that our proposed methods perform well when extending the number and structure of candidate sets and replacing the binary covariate with a continuous one.

AuthorsOpenReview ↗

Response to Reviewer uPhM (Part 3/5)

W3-3. “The proposed method excludes pilot observations from final ATE estimation, potentially sacrificing finite-sample efficiency. Appendix D.3.5 also shows that some full-budget baseline designs outperform EfficientESC at smaller sample sizes.”

Response. Your concern has two components: retaining information already paid for in the pilot and comparing the result with a design that spends the entire budget on the main experiment. We introduce pilot pooling with its covariance-aware variance estimator and evaluate the learned estimators against full-budget balanced designs.

(1) In response to your concern about reusing paid-for pilot observations: Separating design learning from effect estimation makes the main experimental observations conditionally i.i.d. under the learned design (Remark 2, page 6). Section 4.3 (pages 6–7) now introduces EfficientESC+, following sample-size-weighted pooling [8, Section 3.2.2]: τ^+=(1−λP,n)τ^M+λP,nτ^P,,λP,n=nP/(nP+nM).\widehat\tau_+=(1-\lambda_{P,n})\widehat\tau_M +\lambda_{P,n}\widehat\tau_P, \\, \lambda_{P,n}=n_P/(n_P+n_M). Here λP,b\lambda_{P,b} allocates the budget, and λP,n\lambda_{P,n} weights separately fitted estimators using realized sample sizes. Both stage estimators use the same observational sample. Appendix C.2.2 (page 32) estimates the covariance C^O\widehat C_O of their centered observational influence function components ψ^O,AP,ipilot\widehat\psi_{O,A_P,i}^{\mathrm{pilot}} and ψ^O,A^,imain\widehat\psi_{O,\widehat A,i}^{\mathrm{main}}. Equation C.39 (page 33) incorporates 2λP,n(1−λP,n)C^O2\lambda_{P,n}(1-\lambda_{P,n})\widehat C_O in the estimated nn-scaled variance. Corollary 1 (page 8) establishes asymptotic normality and variance consistency under Theorem 2 and Condition B.1. The latter requires consistent pilot nuisances and a pilot product remainder satisfying (nP/n)BP,b=op(1)(n_P/\sqrt n)B_{P,b}=o_p(1). Appendix B.3 (pages 26–27) gives the proof.

(2) In response to your concern about comparison with the full-budget balanced designs: Appendix D.3.4 adds the full-budget comparison in Figure D.4 (page 42). The table below compares the learned estimators with the best prespecified balanced design at the same total budget. Entries are normalized variances.

Methodn=500n=500n=5000n=5000n=10000n=10000
EfficientESC26.33916.66316.695
EfficientESC+20.50815.01815.293
Full-budget balanced 1,2\\{1,2\\}19.73316.56516.752

The revised full-budget comparison quantifies the finite-sample advantages of EfficientESC+ over other methods.

Reviewer_1jzyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors study how to design a randomized experiment that helps estimate the average treatment effect on a main outcome. A large observational dataset records the main outcome, but an estimate from these data alone is selection-biased. The experiment records only short-term secondary outcomes, not the main outcome. A known method, experimental selection correction, uses these secondary outcomes to remove the bias from the observational estimate, but it takes as fixed which secondary outcomes to measure and how to assign treatment. The authors treat both as design choices under a budget limit: more outcomes cost more, so fewer people can be enrolled. The authors pick the design that gives the smallest large-sample variance of the effect estimate. A small scale experiment that measures all candidate outcomes estimates this variance for each design, and a main experiment then uses the chosen design. The authors prove that the chosen design converges to the best one and that the final estimate is approximately normal with the smallest variance among the candidate designs, which gives valid confidence intervals.

Critical strengths
  1. The paper poses a clear and useful new design problem: which secondary outcomes to measure and how to assign treatment, chosen jointly under a budget (Sec. 1, Sec. 3.3 to 3.4). The criterion in Eq. 3 adds the observational variance to the experimental variance scaled by the cost of each design, so the trade-off between information and cost is explicit. This is the main value of the work.

  2. The theory covers the full procedure. Theorem 1 shows that, as the sample grows, the pilot picks the unique best design with probability going to one, and Theorem 2 gives approximate normality at the efficiency bound with a valid confidence interval (Eq. 11, App. B). The proof handles the hard step that the same observational data is used to choose the design and to estimate the effect (App. B.2, Assumption 3(iii)).

Critical weaknesses
  1. The simulation does not show that learning the design is worth it, and the abstract overstates the gains. A simple fixed design that measures the right outcomes and spends the whole budget on the main experiment does better than the proposed method for small samples, and about the same for large samples (App. D.3.5, Fig. D.4). The pilot picks the right outcomes only about 47% of the time at n = 500 (App. D.3.2), and a smaller pilot generally works better (Fig. D.2). Against the strongest baseline the gain is small: about 1 unit out of 17 to 18 in Fig. 1, with no error bars, and 0 to 4 points of power in Table D.1. Sec. 6.2 still calls this "significantly higher". All results come from one small synthetic setup with three outcomes and one binary covariate (App. D.1). The paper uses no real data, although a real example motivates it (Sec. 1).

  2. The new part is small, and the guarantees need strong conditions. The efficiency bound and the estimator come from Chen and Ritzwoller (2023) (Sec. 3.2), and using a pilot to choose how to assign treatment comes from Hahn, Hirano and Karlan (2011). What is new is the budget term in Eq. 3 and choosing the outcomes and the assignment together. The guarantees need one clearly best set of outcomes (Assumption 2(iv)), and a pilot that takes a shrinking share of the budget (Sec. 3.3). They also need the user to supply only outcome sets that already remove the bias (Assumption 1). The paper gives no practical rule for the pilot size, and in the simulation the pilot uses about 38% of the budget at n = 500 (App. D.1.2). The pilot data are also thrown away at the end (Remark 2).

Questions for authors
  1. In App. A.2, the conditioning set is built from the pilot information, which contains all observational data (Sec. 5.2). The proof then treats the observational units as independent given this set and states a conditional central limit theorem with the observational variance. Can you state what is held fixed, and fix this step?

  2. What happens to the estimator and the coverage of the interval in Eq. 11 when two outcome sets tie (so that Assumption 2(iv) fails), or differ by less than the pilot can detect?

  3. Assumption 3(iii) needs a Donsker class (a limit on how complex the fitted models can be), but p. 8 justifies Assumption 3(ii) with forests, boosting, lasso and neural networks, which need not satisfy it unless their complexity is limited. Which nuisance estimators satisfy both conditions, and does the simulation setup do so?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 1jzy (Part 1/4)

Thank you for your constructive comments. We apologize for the delayed response, as we devoted considerable time to extensive revisions prompted by your feedback. We would be grateful for your review of our revised manuscript and responses.

W2-1. “The simulation does not show that learning the design is worth it, and the abstract overstates the gains. [...] The paper uses no real data, although a real example motivates it (Sec. 1).”

Response. We address these concerns with pilot reuse, full-budget comparisons and expanded simulations.

(1) Full-budget comparisons. We introduce EfficientESC+ in Section 4.3 (pages 6–7) and compare it with prespecified balanced designs using the full experimental budget in Appendix D.3.4. Figure D.4 (page 42) shows that EfficientESC+ outperforms all three full-budget balanced designs and closely tracks Oracle+ at larger sample sizes.

The table reports normalized variances.

Methodn=500n=500n=5000n=5000n=10000n=10000
EfficientESC26.33916.66316.695
EfficientESC+20.50815.01815.293
Full-budget balanced 1,2\\{1,2\\}19.73316.56516.752

(2) Design recovery and pilot-budget sensitivity. Here nn is the observational sample size. Figure 2 (page 10) shows that the probability of selecting A∗A^* increases from 46.6% at n=500n=500 to 99.8% at n=5000n=5000. Figures D.1–D.2 (page 40–41) compare five values of the constant factor in λP,b\lambda_{P,b}. Figure D.2 (page 41) shows that EfficientESC+ is insensitive to the constant factor.

(3) Magnitude of gains. Figure 1 (page 10) plots normalized variance, which summarizes the efficiency of each design rather than a point estimate of the ATE. We therefore present the design-comparison curves without error bars. Table D.1 (page 39) reports power of 59.0%, 55.5%, and 50.9% for EfficientESC, Regret-optimal, and Naive at n=5000n=5000 and τ0=2.1\tau_0=2.1, with null sizes of 4.8%, 4.7%, and 4.7%. Figure 1 (page 10) now includes EfficientESC+, and Figure D.4 (page 42) includes EfficientESC+ into comparison. The main comparison measures efficiency under a common pilot deduction, while Figure D.4 evaluates the question of competing with a prespecified design that spends the full budget on final estimation.

(4) Wording. The revised abstract (page 1) describes the observed results as “efficiency gains”, and Section 6.2 (page 9) discusses the actual variance comparisons.

(5) Simulation settings. Appendix D.3.8 (pages 46–48) adds a fourth secondary outcome and evaluates all eight sets containing the identifying outcome, including non-nested candidates. Appendix D.3.9 (pages 48–49) replaces binary XX by X∼Unif[0,1]X\sim\mathrm{Unif}[0,1], allows the outcome coefficients to vary with XX, and defines a two-parameter linear assignment-rule class. At n=5000n=5000, oracle-set recovery is 96.70% and 97.46%, respectively. Figures D.13–D.16 (pages 48–49) report the variance and recovery comparisons.

(6) Real-data evaluation. Section 6 (page 9) explains that the NYC administrative records required by the motivating STAR/NYC application are not publicly available. Applying ESC also requires a suitable pair of observational and experimental data sources with corresponding secondary outcomes satisfying the identification conditions in Assumption 1. We therefore conduct simulation-based experiments for the methodological evaluation of our proposed design.

Simulation-based evaluation is a common approach in methodological studies of experimental design. For example, Aickin [7] evaluates design-adaptive allocation through simulations, Wathen and Thall [8] compare outcome-adaptive randomization procedures with equal randomization in multi-arm clinical trials, and Du et al. [9] compare response-adaptive randomization procedures with equal randomization through simulated trials. These studies evaluate the operating characteristics of competing designs under controlled scenarios without relying on a real-data application.

In our setting, the data-generating process specifies both potential secondary outcomes, Si(1)S_i(1) and Si(0)S_i(0), for every simulated unit ii. This allows us to compare alternative treatment-assignment rules on the same simulated population, with known treatment effects and oracle design criteria. Such a controlled evaluation is particularly useful for assessing design recovery, estimation efficiency, MSE, and confidence-interval coverage. This simulation-based evaluation does not imply that a prospective experiment must observe both potential outcomes for the same individual. Related methodological evaluations also use simulations calibrated to experimental data [1, Section 5] and synthetic case studies based on real-world data [2]. We acknowledge that evaluating our method using suitable empirical datasets is an important direction for future work, as noted in Section 6.

AuthorsOpenReview ↗

Response to Reviewer 1jzy (Part 2/4)

W2-2. “The new part is small, and the guarantees need strong conditions. The efficiency bound and the estimator come from Chen and Ritzwoller (2023) (Sec. 3.2), and using a pilot to choose how to assign treatment comes from Hahn, Hirano and Karlan (2011). What is new is the budget term in Eq. 3 and choosing the outcomes and the assignment together. The guarantees need one clearly best set of outcomes (Assumption 2(iv)), and a pilot that takes a shrinking share of the budget (Sec. 3.3). They also need the user to supply only outcome sets that already remove the bias (Assumption 1). The paper gives no practical rule for the pilot size, and in the simulation the pilot uses about 38% of the budget at n=500n=500 (App. D.1.2). The pilot data are also thrown away at the end (Remark 2).”

Response.

Your comment raises distinct questions about the contribution beyond existing estimation and assignment methods, the scope of the assumptions and the use of the pilot budget. We make the design objective explicit and connect each assumption or budget choice to a corresponding theoretical statement or numerical comparison.

(1) In response to your concern about the methodological contribution: Chen and Ritzwoller [1] provide the efficient estimation framework for combining short- and long-term outcomes under a given design, and Hahn et al. [3] learn treatment assignment from a first wave. The focus of our work is to jointly optimize which outcomes are measured and how treatment is assigned: min⁡(A,ϱA)∈DVO(A)+h,c‾A(ϱA)VE(A,ϱA).\min_{(A,\varrho_A)\in\mathfrak D} \\{V_O(A)+h\\,\overline c_A(\varrho_A)V_E(A,\varrho_A)\\}. Changing AA changes the observational information, experimental variance and affordable experimental sample size. Equations 3–4 (page 5) make this coupling explicit. Theorem 1 (page 8) establishes design recovery, and Theorem 2 (page 8) establishes inference for the learned design.

(2) In response to your concern about the unique-optimum assumption: Assumption 2(iv) identifies a unique oracle outcome set for the recovery theorem. Appendix D.3.6 (pages 45–46) now supplements that result with an exact asymptotic tie between 1,2\\{1,2\\} and 1,2,3\\{1,2,3\\} under a fixed DGP. Figures D.10–D.11 and the bias/coverage table in Q2-2 directly evaluate selection and inference in this setting.

(3) In response to your concern about the identifying requirements: Section 3.1 (page 3) states the identifying conditions required of the prespecified candidate collection. Appendix D.3.5 (pages 42–45) additionally places invalid sets in that collection and evaluates their effect on selection, MSE and coverage. Figures D.8–D.9 (pages 44–45) show how the consequences vary with the experimental budget at fixed n=5000n=5000.

(4) In response to your concern about the finite-budget pilot constant: At n=500n=500 and n/b=0.8n/b=0.8, the default 3Nb−1/33N_b^{-1/3} corresponds to a pilot proportion of approximately 38%. Figures D.1–D.2 (page 40–41) retain this default and compare aP=1,…,5a_P=1,\ldots,5. At this sample size, normalized variance ranges from 22.080 to 32.612 for EfficientESC and from 19.782 to 20.508 for EfficientESC+. The latter comparison directly shows insensitivity to the constant aPa_P.

(5) In response to your concern about discarding pilot outcomes: Section 4.3 (pages 6–7) introduces EfficientESC+ which incorporates the pilot into the final estimation. It uses the stage-pooling construction related to Tabord-Meehan [4, Section 3.2.2]: τ^+=(1−λP,n)τ^M+λP,nτ^P,,λP,n=nP/(nP+nM).\widehat\tau_+=(1-\lambda_{P,n})\widehat\tau_M+\lambda_{P,n}\widehat\tau_P, \\, \lambda_{P,n}=n_P/(n_P+n_M). The pilot estimator is defined by Equation C.38 (page 32). Appendix C.2.2 (page 32) describes its final cross-fitting. Corollary 1 (page 8) establishes asymptotic normality and variance consistency under Theorem 2 and Condition B.1, with the proof in Appendix B.3 (pages 26–27).

AuthorsOpenReview ↗

Response to Reviewer 1jzy (Part 3/4)

Q2-1. “In App. A.2, the conditioning set is built from the pilot information, which contains all observational data (Sec. 5.2). The proof then treats the observational units as independent given this set and states a conditional central limit theorem with the observational variance. Can you state what is held fixed, and fix this step?”

Response. You identify an important conditioning issue: once the observational sample is included in the conditioning information, it cannot supply a nondegenerate conditional observational CLT. The revised proof uses an unconditional observational CLT and a conditional main-experiment CLT, combined through iterated expectations.

(1) In response to your concern about what conditioning holds fixed: Section 5.2 (page 8) defines

Fpilot=σ(DO,DEX,DP,Upilot).\mathcal F_{\mathrm{pilot}} = \sigma(\mathcal D_O,\mathcal D_E^X, \mathcal D_P,\mathcal U_{\mathrm{pilot}}).

Conditioning fixes the observational sample, pilot data, cost-estimation covariates, learned design, and nMn_M. The observational contribution therefore has conditional variance zero, while the main experimental observations remain conditionally i.i.d. under the learned design.

(2) In response to your concern about the corrected CLT argument: The revised proofs in Appendices A.2 (pages 16–20) and Lemma B.10 (page 25) distinguish between the observational and experimental contributions. The former is handled by an unconditional CLT, while the latter is handled by a conditional CLT given the pilot-stage information.

For Theorem 2 (page 8), by equation (B.24), the two leading terms are Un=∑i∈IOφO,i/n,Tn=n∑j∈IMφE,j(ϱ^A∗)/nM.U_n=\sum_{i\in\mathcal I_O}\varphi_{O,i}/\sqrt n, \qquad T_n=\sqrt n \sum_{j\in\mathcal I_M} \varphi_{E,j}(\widehat\varrho_{A^*})/n_M. The observational term satisfies Un⟶dN(0,VO(A∗))U_n\overset{d}{\longrightarrow}N(0,V_O(A^*)) by the unconditional Lyapunov CLT. Although UnU_n is Fpilot\mathcal F_{\mathrm{pilot}}-measurable, no conditional CLT is needed for this term. In contrast, conditional on Fpilot\mathcal F_{\mathrm{pilot}}, ϱ^A∗\widehat\varrho_{A^*} and nMn_M are fixed, and the main experimental observations remain i.i.d. By Lemmas B.7 and B.10, for every t∈Rt\in\mathbb R, E[eitTn∣Fpilot]⟶pe−t2vE/2,,vE=hcA∗(ϱA∗∗)VE(A∗,ϱA∗∗).\mathbb E\left[e^{itT_n}\mid\mathcal F_{\mathrm{pilot}}\right] \overset{p}{\longrightarrow}e^{-t^2v_E/2},\\,v_E=h c_{A^*}(\varrho_{A^*}^*) V_E(A^*,\varrho_{A^*}^*). Since the conditional characteristic functions are bounded by one, the convergence also holds in L1L^1.

Finally, since UnU_n is Fpilot\mathcal F_{\mathrm{pilot}}-measurable, iterated expectations yield E[eit(Un+Tn)]=E[eitUnE(eitTn∣Fpilot)]⟶e−t2VO(A∗)+vE/2.\mathbb E[e^{it(U_n+T_n)}] =\mathbb E\left[ e^{itU_n} \mathbb E(e^{itT_n}\mid\mathcal F_{\mathrm{pilot}}) \right] \longrightarrow e^{-t^2\\{V_O(A^*)+v_E\\}/2}. Thus, the two contributions jointly yield the unconditional asymptotic normality in Theorem 2, without requiring their independence.

Q2-2. “What happens to the estimator and the coverage of the interval in Eq. 11 when two outcome sets tie (so that Assumption 2(iv) fails), or differ by less than the pilot can detect?”

Response.

Your question concerns inference when set recovery does not reduce to a unique optimum. We add an exact asymptotic-tie experiment.

(1) In response to your concern about selection when two sets are optimal: Appendix D.3.6 (page 45) makes 1,2\\{1,2\\} and 1,2,3\\{1,2,3\\} jointly optimal under the asymptotic criterion σ0.82\sigma_{0.8}^2 by adding γS3(w)−S2(w)\gamma\\{S_3(w)-S_2(w)\\} to both potential primary outcomes. The same DGP is used for every nn, and the true ATE is unchanged. Figure D.10 (page 45) reports selection of each optimum and their union: the union probability increases from 39.35% at n=500n=500 to 99.05% at n=5000n=5000 and 99.98% at n=10000n=10000. Asymptotic equality does not require equal selection frequencies, and pilot-budget deduction can distinguish the two finite-budget criteria.

(2) In response to your concern about the estimator variance in the tie setting: Figure D.11 (page 46) compares normalized variances. At n=5000n=5000, EfficientESC+ achieves 20.693 versus 23.615 for EfficientESC and 20.619 for Oracle+.

(3) In response to your concern about coverage and the interpretation of near ties: The table reports normalized variance, bias and empirical coverage in the exact-tie setting.

nnMethodNormalized variance10310^3 BiasCoverage (%)
500EfficientESC37.6372.30193.5
500EfficientESC+26.569-1.62894.3
5000EfficientESC23.6150.99195.2
5000EfficientESC+20.6930.79395.3
10000EfficientESC23.667-0.15994.7
10000EfficientESC+21.607-0.21694.8

The table, together with Figures D.10 and D.11, demonstrates that the proposed methods can retain their performance even when Assumption 2 (iv) is violated.

Reviewer_BfXwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EfficientESC, which uses a pilot experiment to select secondary outcomes and treatment-assignment probabilities under a budget constraint. The goal is to improve ATE estimation by combining observational data with experiments that do not measure the primary outcome. The authors establish oracle-design recovery and asymptotic efficiency, and evaluate the method through simulations.

Critical strengths
  1. Jointly choosing which outcomes to measure and how to assign treatment is a meaningful contribution. The framework directly accounts for the tradeoff between measurement costs and estimation precision.

  2. The theory covers both design learning and final inference. Simulations examine design recovery, variance, and test calibration. The full-budget comparison also provides a useful assessment of the pilot’s cost.

Critical weaknesses

Every candidate outcome set must satisfy the identification assumptions. The method selects among valid sets but cannot establish their validity. The simulation makes this condition particularly favorable. All candidate sets include an outcome that reveals the confounder given treatment. Performance with imperfect proxies remains unclear.

Questions for authors
  1. If some candidate outcome sets slightly violate latent unconfoundedness, could variance-based selection favor a biased design? How would this affect MSE and confidence-interval coverage? Could the authors discuss a bias-aware selection criterion, in relation to Rothenhäusler’s Model selection and inference for estimation of causal parameters (2024)?

  2. The paper derives an asymptotic rate for the pilot budget proportion, but how should its constant be chosen for a given total budget? Appendix D.3.3 suggests that smaller pilot proportions generally perform better in the tested range. What guidance does the method provide for balancing design-learning accuracy against the sample size of the main experiment?

3 comments
AuthorsOpenReview ↗

Response to Reviewer BfXw (Part 1/2)

Thank you for your constructive comments. We apologize for the delayed response, as we devoted considerable time to extensive revisions prompted by your feedback. We would be grateful for your review of our revised manuscript and responses.

W1. “Every candidate outcome set must satisfy the identification assumptions. The method selects among valid sets but cannot establish their validity. The simulation makes this condition particularly favorable. All candidate sets include an outcome that reveals the confounder given treatment. Performance with imperfect proxies remains unclear.”

Response. Your concern distinguishes the validity of a secondary-outcome set from the efficiency of a design based on that set. We have clarified the identification requirement and expanded the simulations where invalid candidate sets are included.

(1) In response to your concern about the validity of candidate outcome sets: Latent Unconfoundedness (LU) is the identifying restriction inherited from ESC [1]. Section 3.1 (page 3) clarifies that the candidate collection is prespecified using substantive identification arguments. Our focus here is experimental design, not identification. A related distinction arises in instrumental-variable research: substantive arguments motivate the validity of candidate instruments, while statistical methods can select among candidate instruments but do not, by themselves, establish their validity. Angrist and Krueger [2] motivate quarter of birth as an instrument through compulsory schooling laws, Card [3] use college proximity as an instrument, and Belloni et al. [4] develop statistical methods for selecting instruments.

(2) In response to your concern about the favorable baseline DGP and a broader stress test: In our DGP, S1(w)=U+0.5wS_1(w)=U+0.5w supplies identification, S2S_2 adds prognostic information, and S3S_3 is redundant conditional on S2S_2. Adding the latter outcomes preserves LU because (V,Z)⊥!!!⊥W∣(U,X)(V,Z)\perp\\!\\!\\!\perp W\mid(U,X). Section 6.1 (page 9) explains these roles, and Appendix D.3.5 (pages 42–45) additionally examines candidates excluding S1S_1. The revision therefore makes the validity requirement explicit and supplements the baseline with a direct assessment of what happens when that requirement is violated.

Q1-1. “If some candidate outcome sets slightly violate latent unconfoundedness, could variance-based selection favor a biased design? How would this affect MSE and confidence-interval coverage? Could the authors discuss a bias-aware selection criterion, in relation to Rothenhäusler's Model selection and inference for estimation of causal parameters (2024)?”

Response. Your question concerns both the risk of biased selection and its consequences for estimation and inference. We now evaluate these consequences directly through a new experiment including both valid and invalid secondary outcome sets.

(1) In response to your concern about biased selection under invalid candidates: A criterion based only on variance can favor an estimator with identification bias. Appendix D.3.5 therefore evaluates all seven nonempty outcome sets, including the three invalid sets 2\\{2\\}, 3\\{3\\}, and 2,3\\{2,3\\}. At n/b=0.1n/b=0.1, Figure D.6 (page 43) shows recovery of the valid oracle 1,2\\{1,2\\} reaching 99.98% at n=10000n=10000.

(2) In response to your concern about the resulting MSE and confidence-interval coverage: We fix n=5000n=5000 and vary the experimental budget. Figures D.8–D.9 (pages 44–45) report MSE and coverage, and the table below gives results at three budgets. Pilot reuse reduces MSE at each reported budget.

b/nb/n103MSE10^3\mathrm{MSE} (EfficientESC)103MSE10^3\mathrm{MSE} (EfficientESC+)Coverage (%) (EfficientESC)Coverage (%) (EfficientESC+)
1498.256357.3800.000.00
5190.167157.68758.458.7
105.8905.24793.593.8

(3) In response to your concern about a bias-aware selection criterion: Rothenhäusler [5] compares potentially biased estimators with a reliable benchmark and corrects estimated risk for sampling variability. However, extending Rothenhäusler's theory requires accounting for the slower-growing pilot sample, dependence among candidate estimators due to their shared observational sample, and uncertainty from design selection. It also requires a substantively justified valid benchmark. We view this as a promising direction for extending EfficientESC to settings with potentially invalid candidate sets.

AuthorsOpenReview ↗

Response to Reviewer BfXw (Part 2/2)

Q1-2. “The paper derives an asymptotic rate for the pilot budget proportion, but how should its constant be chosen for a given total budget? Appendix D.3.3 suggests that smaller pilot proportions generally perform better in the tested range. What guidance does the method provide for balancing design-learning accuracy against the sample size of the main experiment?”

Response. Your question asks how to use the result of pilot proportions to balance design-learning accuracy against the sample size of the main experiment. We distinguish the rate-balancing calculation from its leading constant and use the new experiment reusing the pilot to assess sensitivity to that constant.

(1) In response to your concern about the pilot-budget rate and its leading constant:

Remark 1 (page 5) and Appendix C.1 (pages 27–30) derive the allocation rate by balancing the budget reserved for the pilot against criterion-estimation error. Equation C.34 (page 29) defines the proxy, and Equation C.35 gives its general solution. For root-sample-size criterion error, γP=1/2\gamma_P=1/2, the specialization on page 30 is

λP,b∗=(CP2CF)2/3Nb−1/31+o(1).\lambda_{P,b}^{*} =\left(\frac{C_P}{2C_F}\right)^{2/3}N_b^{-1/3}\\{1+o(1)\\}.

The Nb−1/3N_b^{-1/3} order follows from balancing the two terms, while the leading multiplier depends on their constants. The sensitivity experiment assesses the multiplier.

(2) Some related work: Hahn et al. [6] and Tabord-Meehan [7] use a first wave to learn later treatment assignment. Viviano [8] studies pilot-based two-wave design under network interference, while Cai and Rafi [9] analyze the estimation error of feasible Neyman allocation with small pilots. These papers motivate examining pilot precision at finite budgets, where the constants and trade-offs depend on their respective design problems.

(3) In response to your concern about finite-budget guidance from the sensitivity experiment: We retain aP=3a_P=3 as the default and evaluate λP,b=aPNb−1/3\lambda_{P,b}=a_PN_b^{-1/3} for aP∈1,2,3,4,5a_P\in\\{1,2,3,4,5\\} in Figures D.1–D.2 (pages 40–41). Figure D.2 (page 41) and the table below show that EfficientESC+ is insensitive to the constant factor in λP,b\lambda_{P,b}.

MethodaP=1a_P=1aP=2a_P=2aP=3a_P=3aP=4a_P=4aP=5a_P=5
EfficientESC22.08023.90326.33929.33032.612
EfficientESC+20.42120.15120.50819.78219.827

The entries are normalized variances at n=500n=500.

References

[1] Athey, S., Chetty, R., and Imbens, G. (2025). The Experimental Selection Correction Estimator: Using Experiments to Remove Biases in Observational Estimates. NBER Working Paper No. 33817.

[2] Angrist, J. D., and Krueger, A. B. (1991). Does Compulsory School Attendance Affect Schooling and Earnings? The Quarterly Journal of Economics, 106(4), 979–1014.

[3] Card, D. (1995). Using Geographic Variation in College Proximity to Estimate the Return to Schooling. In Aspects of Labor Market Behaviour: Essays in Honour of John Vanderkamp, pp. 201–222. University of Toronto Press.

[4] Belloni, A., Chen, D., Chernozhukov, V., and Hansen, C. (2012). Sparse Models and Methods for Optimal Instruments with an Application to Eminent Domain. Econometrica, 80(6), 2369–2429.

[5] Rothenhäusler, D. (2024). Model Selection and Inference for Estimation of Causal Parameters. Electronic Journal of Statistics, 18(2), 5449–5483.

[6] Hahn, J., Hirano, K., and Karlan, D. (2011). Adaptive Experimental Design Using the Propensity Score. Journal of Business & Economic Statistics, 29(1), 96–108.

[7] Tabord-Meehan, M. (2023). Stratification Trees for Adaptive Randomisation in Randomised Controlled Trials. The Review of Economic Studies, 90(5), 2646–2673.

[8] Viviano, D. (2022). Experimental Design under Network Interference. Working Paper, July 2022 version. arXiv:2003.08421.

[9] Cai, Y., and Rafi, A. (2024). On the Performance of the Neyman Allocation with Small Pilots. Journal of Econometrics, 242(1), Article 105793.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer BfXw,

I hope you are doing well! As the discussion period is drawing to a close, we would be very grateful if you could let us know whether our responses and additional experiments have adequately addressed your concerns.

Thank you again for your valuable feedback, time, and thoughtful consideration.

Best regards,

Authors of Submission61497 (EfficientESC)

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of the revisions

We thank the reviewers for their time and constructive feedback on our paper, which develops a novel framework that learns the optimal design for using experimentation to correct selection bias in observational studies. We apologize for the delayed response, as we devoted considerable time to extensive revisions prompted by the reviewers' constructive comments. We would be grateful if the reviewers could examine our revised manuscript and responses.

All three reviewers agree on the value of jointly choosing which secondary outcomes to measure and how to assign treatment.

  • Reviewer BfXw identifies this joint choice as a meaningful contribution and emphasizes the explicit trade-off between measurement cost and estimation precision.

  • Reviewer 1jzy describes it as a clear and useful new design problem, and identifies the variance criterion as the main value of the work, which makes the trade-off between information and cost explicit.

  • Reviewer uPhM recognizes the importance of the problem and the framework's principled treatment of measurement cost and statistical efficiency.

Theoretical contributions are widely acknowledged.

  • Reviewer BfXw values the analysis of both design learning and final inference, and notes that the full-budget comparison usefully assesses the pilot's cost.

  • Reviewer 1jzy highlights oracle-design recovery, efficient inference, and the treatment of observational-data reuse.

  • Reviewer uPhM recognizes the guarantees for design recovery, asymptotic normality, and semiparametric efficiency.

The reviewers raised concerns about the finite-sample performance and practical benefits of the proposed method, the limited scope of the simulation studies, the role and robustness of the identification assumptions, and the practical implications of the cost and budget constraints. We have thoroughly addressed these in our rebuttal. A summary of the revisions is as follows.

Methodological and theoretical advances. We propose a new method termed EfficientESC+, which incorporates pilot sample into final effect estimation without additional experimental expenditure or changes to the learned main-stage design (pp. 6–7 of the revised paper). By combining the original main-stage estimator (i.e., the previous EfficientESC) with a pilot-stage estimator, EfficientESC+ improves finite-sample efficiency while retaining the same first-order asymptotic efficiency, as established in our new Corollary 1 (p. 8, with proof on pp. 26–27), showing that it attains the same minimized semiparametric efficiency bound. Extensive simulations demonstrate that EfficientESC+ achieves greater estimation efficiency than EfficientESC, and outperforms the other competing methods (pp. 9–10 and pp. 41–42). Moreover, we have also clarified the identification requirements (p. 3), refined the pilot-budget analysis (p. 5 and pp. 27–30), and strengthened the conditioning and nuisance-estimation arguments (p. 8 and pp. 23–25).

Numerical experiments. We have expanded the simulation study to include (i) EfficientESC+ in the main and full-budget comparisons (pp. 9–10 and pp. 41–42), (ii) pilot budget proportion sensitivity analyses for EfficientESC+ (pp. 40–41), (iii) invalid candidate outcome sets, with MSE and coverage evaluated across experimental budgets (pp. 42–45), and (iv) an exact asymptotic tie between two optimal outcome sets (pp. 45–46). We additionally examine (v) three cost and budget settings that distinguish known conditional costs from estimated average costs and expected-budget constraints from strict-budget constraints (pp. 45–47). Finally, we add (vi) a setting with four secondary outcomes and eight candidate sets (pp. 46–48) and (vii) a setting with a continuous covariate (pp. 48–49).

We have carefully revised the manuscript to incorporate the corresponding improvements, with all modifications highlighted in red. We sincerely appreciate the insightful feedback and believe that these revisions significantly strengthen the paper.

AuthorsOpenReview ↗

Future directions

In response to the reviewers' additional comments, we have rewritten the Discussion section in Appendix E (pp. 50-51) to clarify three limitations and outline how future work could address them.

  • Identification. Our theory requires latent unconfoundedness for the candidate set of secondary outcomes. Future work could develop novel sensitivity analysis and partial identification into the ESC framework, or study identification using instrumental variables or additional proxies.

  • Real-world evaluation. Our numerical evaluation relies on synthetic data. Future work could evaluate the design using simulations calibrated to real data or through real field experiments.

  • Pilot reuse. EfficientESC+ improves efficiency in our simulations, but it has the same first-order asymptotic variance as EfficientESC because the pilot budget proportion vanishes. Future work could theoretically quantify this gain through higher-order analysis and develop finite-sample guarantees for design selection.

These questions are beyond the scope of the current paper, and we plan to explore them in future work. We thank the reviewers for helping us identify these important research directions.

AuthorsOpenReview ↗

Final comment from authors

We want to thank each reviewer once again for the time and effort devoted to our paper and for the valuable feedback throughout the rebuttal. We are pleased that the reviewers found the problem important and saw value in our approach to seeking the optimal design for experimental selection correction (ESC). Following the rebuttal process, we summarize the major revisions to our paper as follows.

  1. Reusing the pilot in effect estimation. Following suggestions from Reviewers 1jzy and uPhM, we introduced EfficientESC+ in Section 4.3. It uses the randomized pilot both to learn the design and to estimate the primary average treatment effect, at no additional experimental cost. Corollary 1 establishes that it attains the minimized semiparametric efficiency bound. The simulations illustrate its clear benefit.

  2. Strengthening the theoretical arguments. In response to Reviewer 1jzy, we corrected the conditioning argument in the proof of Theorem 2. Reviewer 1jzy confirmed that this revision resolved the main technical concern. We also clarified the conditions on nuisance estimation and, following Reviewer BfXw’s question about pilot sample size, refined the derivation of the pilot budget proportion.

  3. Conducting additional numerical experiments. Questions from Reviewers BfXw and uPhM motivated a new experiment that quantifies the effects of invalid candidate outcome sets on estimation error. Following Reviewer 1jzy’s suggestion, we also examined a setting in which two outcome sets are equally optimal. In response to Reviewer uPhM, we considered a larger collection with non-nested outcome sets and evaluated the procedure under estimated measurement costs and strict budget constraints (Appendix D.3).

  4. Rewriting the discussion section. The follow-up comments from Reviewers 1jzy and uPhM prompted us to rewrite the discussion section in Appendix E (pp. 50–51). We now explain how sensitivity analysis and partial identification could extend ESC when latent unconfoundedness fails. We also outline how a prospective field experiment could evaluate the proposed design in practice and how higher-order theory could characterize the theoretical gains observed from pilot reuse.

We thank the Area Chair in advance for considering the revised paper and the discussion. We believe our paper makes an impactful contribution because it extends experimental selection correction, which is an important and influential method in the fields of causal inference and econometrics, from estimating a treatment effect under a given design to learning which experiment to run. We show how to use the semiparametric efficiency bound to guide data collection and prove that a feasible two-stage procedure attains the bound of the oracle design. The remaining questions from reviewers concern extensions of this result. Each would warrant a separate study, as summarized in our discussion section.

We hope all of us have fun doing research and pursuing topics that genuinely interest us in this era of AI.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.