- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EfficientESC, a two-stage experimental design framework that jointly optimizes secondary outcome selection and treatment allocation under budget constraints. The authors characterize an oracle design that minimizes the semiparametric efficiency bound for average treatment effect (ATE) estimation, and establish theoretical guarantees for design recovery and valid inference. Simulation results show that the proposed method achieves efficiency improvements relative to several competing methods.
Critical strengths
-
The paper addresses an important experimental design problem by jointly optimizing secondary outcome selection and treatment allocation under budget constraints.
-
The proposed framework explicitly accounts for the trade-off between measurement cost and statistical efficiency, providing a principled approach to experimental design.
-
The theoretical analysis establishes oracle design recovery, asymptotic normality, and semiparametric efficiency under the stated assumptions.
Critical weaknesses
-
The framework assumes that all candidate secondary outcome sets satisfy latent unconfoundedness, which is difficult to verify in practice and may significantly limit its applicability.
-
The experimental evaluation relies entirely on synthetic data and includes only three nested secondary outcome sets, so its generalizability to more complex real-world settings has not been fully established.
-
The proposed method excludes pilot observations from final ATE estimation, potentially sacrificing finite-sample efficiency. Appendix D.3.5 also shows that some full-budget baseline designs outperform EfficientESC at smaller sample sizes.
-
The framework assumes that measurement costs are known and considers an expected budget constraint, which limits its practical applicability when costs are uncertain or when a strict budget limit must be satisfied.
Questions for authors
-
How robust is EfficientESC when some candidate secondary outcome sets violate the latent unconfoundedness assumption?
-
Can the authors provide additional experiments involving more secondary outcomes, complex covariates, or real-world datasets to demonstrate the generalizability of the method?
-
Would incorporating pilot observations into the final ATE estimation improve finite-sample efficiency, especially when the experimental budget is limited?
-
How can the proposed framework be extended to handle uncertain measurement costs and strict finite-sample budget constraints?
7 comments
Response to Reviewer uPhM (Part 2/5)
W3-2. “The experimental evaluation relies entirely on synthetic data and includes only three nested secondary outcome sets, so its generalizability to more complex real-world settings has not been fully established.”
Response. Your concern asks for evaluation beyond three nested outcome sets and a binary covariate, as well as a clearer account of the real-data motivation. The revision adds a fourth outcome with non-nested candidates and a continuous-covariate setting.
(1) In response to your concern about the motivating application and required data sources: Section 6 (page 9) explains that the NYC administrative records required by the motivating STAR/NYC application are not publicly available. Moreover, applying ESC requires observational and experimental data sources with corresponding secondary outcomes satisfying the identification conditions in Assumption 1. The current study therefore focuses on simulation-based methodological evaluation. We acknowledge the importance of empirical validation using suitable datasets and identify it as a direction for future work in Section 6.
(2) In response to your concern about the role of synthetic data: Simulation allows us to compare alternative treatment-assignment rules under controlled data-generating processes. For each simulated unit , we specify both potential secondary outcomes, and . In observed data, only the potential outcome corresponding to the treatment actually received is observed for a given individual [5]. Specifying both potential outcomes in the simulation therefore allows us to evaluate alternative assignment rules counterfactually on the same underlying population and compare their performance with known treatment effects and oracle design criteria. As illustrated by the methodological studies discussed in W2-1(6), simulation-based evaluation is commonly used in adaptive experimental design. Related evaluations also include simulations calibrated to experimental data [6, Section 5] and synthetic case studies based on real-world data [7].
(3) In response to your concern about more outcomes and continuous covariates: Our additions directly test the design-learning procedure with a larger, non-nested candidate collection and a continuous covariate. Appendix D.3.8 (pages 46–48) adds and compares all eight sets in , including non-nested candidates. Appendix D.3.9 (pages 48–49) uses , covariate-dependent outcome coefficients and the prespecified class
Binary response nuisances use logistic regression, and continuous response nuisances use natural cubic splines with four basis functions. Figures D.13–D.14 (page 48) and D.15–D.16 (page 49) report variance and recovery. The table below adds coverage at .
| Setting | Recovery (%) | Normalized variance () | Normalized variance () | Coverage (%) () | Coverage (%) () |
|---|---|---|---|---|---|
| Four outcomes | 96.70 | 19.630 | 17.871 | 94.7 | 94.5 |
| Continuous | 97.46 | 17.699 | 16.085 | 94.8 | 94.8 |
The added experiments show that our proposed methods perform well when extending the number and structure of candidate sets and replacing the binary covariate with a continuous one.
Response to Reviewer uPhM (Part 3/5)
W3-3. “The proposed method excludes pilot observations from final ATE estimation, potentially sacrificing finite-sample efficiency. Appendix D.3.5 also shows that some full-budget baseline designs outperform EfficientESC at smaller sample sizes.”
Response. Your concern has two components: retaining information already paid for in the pilot and comparing the result with a design that spends the entire budget on the main experiment. We introduce pilot pooling with its covariance-aware variance estimator and evaluate the learned estimators against full-budget balanced designs.
(1) In response to your concern about reusing paid-for pilot observations: Separating design learning from effect estimation makes the main experimental observations conditionally i.i.d. under the learned design (Remark 2, page 6). Section 4.3 (pages 6–7) now introduces EfficientESC+, following sample-size-weighted pooling [8, Section 3.2.2]: Here allocates the budget, and weights separately fitted estimators using realized sample sizes. Both stage estimators use the same observational sample. Appendix C.2.2 (page 32) estimates the covariance of their centered observational influence function components and . Equation C.39 (page 33) incorporates in the estimated -scaled variance. Corollary 1 (page 8) establishes asymptotic normality and variance consistency under Theorem 2 and Condition B.1. The latter requires consistent pilot nuisances and a pilot product remainder satisfying . Appendix B.3 (pages 26–27) gives the proof.
(2) In response to your concern about comparison with the full-budget balanced designs: Appendix D.3.4 adds the full-budget comparison in Figure D.4 (page 42). The table below compares the learned estimators with the best prespecified balanced design at the same total budget. Entries are normalized variances.
| Method | |||
|---|---|---|---|
| EfficientESC | 26.339 | 16.663 | 16.695 |
| EfficientESC+ | 20.508 | 15.018 | 15.293 |
| Full-budget balanced | 19.733 | 16.565 | 16.752 |
The revised full-budget comparison quantifies the finite-sample advantages of EfficientESC+ over other methods.
Response to Reviewer uPhM (Part 1/5)
Thank you for your constructive comments. We apologize for the delayed response, as we devoted considerable time to extensive revisions prompted by your feedback. We would be grateful for your review of our revised manuscript and responses.
W3-1. “The framework assumes that all candidate secondary outcome sets satisfy latent unconfoundedness, which is difficult to verify in practice and may significantly limit its applicability.”
Response. Your concern is about how candidate-set validity enters the design problem. We clarify the substantive identification requirement, explain the roles of the baseline outcomes and add a simulation in which the learner is also offered invalid candidates.
(1) In response to your concern about the identification requirement: Latent Unconfoundedness (LU) is an identifying assumption inherited from ESC [1]. As clarified in Section 3.1 (page 3), the candidate collection is prespecified based on substantive knowledge and is assumed to contain only secondary-outcome sets satisfying the identification conditions. Our framework takes the validity of these candidates as an input and focuses on selecting among them to optimize statistical efficiency, rather than testing or establishing their validity from the data. A related distinction arises in instrumental-variable research: substantive arguments motivate the validity of candidate instruments, while statistical procedures can select among candidates but do not, by themselves, establish their validity. Angrist and Krueger [2] motivate quarter of birth as an instrument through compulsory schooling laws, Card [3] use college proximity as an instrument, and Belloni et al. [4] develop statistical methods for selecting instruments. To examine the consequences of violating the prespecification requirement, Appendix D.3.5 (pages 42–45) additionally includes LU-invalid candidate sets and evaluates design recovery, MSE, and confidence-interval coverage (see Q3-1 below).
(2) In response to your concern about the roles of the simulated secondary outcomes: In our DGP, S1 supplies the identifying information, S2 adds prognostic information, and S3 is redundant conditional on S2. Appendix D.3.5 (pages 42–45) additionally examines invalid candidates, with results reported in Q3-1 below.