Risk-aware A/B testing through efficient inference on treatment effect distributions without a margin assumption

In many fields, including healthcare, marketing, and online platform design, A/B tests are used to evaluate new treatments and make launch decisions based on average treatment effect (ATE) estimates. But ATE-based decision-making overlooks distributional risks, such as a large fraction of individuals affected negatively by the treatment. These distributional risks can be detected by estimating the entire cumulative distribution function of the treatment effect, rather than only its mean. Prior work on estimating such treatment effect distributions has estimated partial identification bounds known as Makarov bounds under restrictive assumptions on the outcome distribution. In particular, these assumptions require that the treatment effect cannot be constant for any subset of experimental units, an assumption that is often either violated or nearly-violated in practice. In this paper, we propose a novel method for asymptotically optimal estimation of and statistically-valid inference on Makarov bounds under more realistic assumptions than existing methods. Our main technical contributions are to a. propose smoothed surrogates for the Makarov bounds, b. derive semiparametrically efficient estimators of these surrogates, and c. propose a procedure for optimal selection of the smoothing parameters. We show empirically on synthetic and semi-synthetic datasets that our estimators achieve a better bias-variance trade-off and lower mean-squared error than existing estimators. Finally, we deploy our method on real A/B test data from a large social media platform, and show how estimates of the treatment effect distribution can inform decision-making.