acceptodds
Under review as a conference paper at ICLR 2027

Risk-aware A/B testing through efficient inference on treatment effect distributions without a margin assumption

Abstract

In many fields, including healthcare, marketing, and online platform design, A/B tests are used to evaluate new treatments and make launch decisions based on average treatment effect (ATE) estimates. But ATE-based decision-making overlooks distributional risks, such as a large fraction of individuals affected negatively by the treatment. These distributional risks can be detected by estimating the entire cumulative distribution function of the treatment effect, rather than only its mean. Prior work on estimating such treatment effect distributions has estimated partial identification bounds known as Makarov bounds under restrictive assumptions on the outcome distribution. In particular, these assumptions require that the treatment effect cannot be constant for any subset of experimental units, an assumption that is often either violated or nearly-violated in practice. In this paper, we propose a novel method for asymptotically optimal estimation of and statistically-valid inference on Makarov bounds under more realistic assumptions than existing methods. Our main technical contributions are to a. propose smoothed surrogates for the Makarov bounds, b. derive semiparametrically efficient estimators of these surrogates, and c. propose a procedure for optimal selection of the smoothing parameters. We show empirically on synthetic and semi-synthetic datasets that our estimators achieve a better bias-variance trade-off and lower mean-squared error than existing estimators. Finally, we deploy our method on real A/B test data from a large social media platform, and show how estimates of the treatment effect distribution can inform decision-making.

Primary area:
causal reasoning
Keywords:
A/B testing, causal inference, treatment effect estimation, double machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TCUhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper develops inference for Makarov bounds on treatment effect distributions without a margin assumption. Smoothing the nonsmooth operators removes the need for unique optimizers, while influence function corrections yield efficient, asymptotically normal estimators of the smoothed bounds for fixed smoothing parameters. One-sided corrections for smoothing bias provide asymptotically conservative confidence intervals for the treatment effect distribution. It also proposes selecting smoothing parameters from the data. Simulations show lower MSE than the baselines, and real A/B tests illustrate how the bounds can inform launch decisions.

Critical strengths

The paper extends Makarov bound inference to settings with nonunique optimizers, where standard envelope guarantees no longer apply. For fixed smoothing parameters and a fixed evaluation threshold, explicit one-sided corrections yield asymptotically conservative confidence intervals for the original treatment effect CDF under the stated regularity conditions.

Critical weaknesses

Data-driven smoothing parameter selection is one of the main contributions, but the current analysis does not justify the claim that this selection is optimal.

  • Equation (16) in Section 4.3 replaces the squared smoothing bias in Equation (15) with b(t1,t2)2b(t_1,t_2)^2. However, Equation (6) in Lemma 4.1 only establishes ρt1,t2−−ρ−≤b(t1,t2)\rho^-_{t_1,t_2}-\rho^-\le b(t_1,t_2), not a bound on the absolute or squared bias. Thus, Equation (6) does not justify this replacement.

  • The Lepski rule in Section 4.3 ranks candidates by b(t1,t2)b(t_1,t_2) in Equation (6), with comparison standard errors given by Equation (17). When ∣Y∣=1|\mathcal Y|=1, b=log⁡2/t2b=\log 2/t_2 ignores t1t_1, although t1t_1 controls the supremum approximation in Equation (4). Thus, this ordering does not generally reflect the candidates' approximation errors.

Questions for authors
  1. Should the variance term in Equation (16) be n−2∑iΨ^i2n^{-2}\sum_i\widehat\Psi_i^2 rather than n−1∑iΨ^i2n^{-1}\sum_i\widehat\Psi_i^2 to match Equation (15)? Why do the risk bounds in Table 1 differ from those in the text (0.370.37 versus 0.250.25, and 0.760.76 versus 0.740.74)? What acceptable risk threshold supports the conclusion that a treatment is safe to launch?

  2. The efficiency results concern smoothed bounds with fixed smoothing parameters. What theoretical guarantee supports the claimed optimality for estimating the original bounds after data-driven tuning?

  3. Does Appendix C recover sharp bounds for discrete outcomes? Without covariates, consider identical Bernoulli$(1/2)$ marginals at δ=0\delta=0. All candidate differences used in Equation (64) are zero, so Equation (67) converges to [0,1][0,1] as t1,t2→∞t_1,t_2\to\infty. However, P(Y(1)≤Y(0))≥P(Y(1)=0)=1/2P(Y(1)\le Y(0))\ge P(Y(1)=0)=1/2, and the sharp bounds are [1/2,1][1/2,1]. Thus, the concern is not that the bounds are invalid, but that they fail to recover the claimed sharp bounds. How should ties be handled, and would correcting the formulas in Appendix C affect the TWINS results in Figure 6?

Reviewer_xp22OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies distributional risks in A/B testing through bounds on the individual treatment effect distribution. It introduces smoothed Makarov bounds, derives efficient influence functions and debiased estimators, and constructs confidence intervals without a margin assumption. Synthetic, semi-synthetic, and three real-world A/B experiments evaluate the approach and illustrate its potential for risk-aware decisions.

Critical strengths
  1. The paper addresses an important practical limitation of average treatment effects: positive average gains can coexist with harm to individual users.

  2. The combination of smooth approximations, influence-function-based estimation, and confidence corrections provides a useful framework for inference without a margin assumption.The evaluation includes synthetic, semi-synthetic, and large-scale real-world experiments. Reporting results over 2,500 simulation repetitions strengthens the empirical evidence.

Critical weaknesses
  1. Discrete extension. Appendix C does not appear to recover sharp lower bounds. If both potential outcomes have Bernoulli(0.5) marginals and δ=0\delta = 0, its same-sided CDF differences are zero, whereas the sharp lower bound on P(Y(1)−Y(0)≤0)P(Y(1)-Y(0)\leq 0) is 0.50.5. This matters for the paper’s discrete-outcome applications.Tuning criterion. Equation (16) is insufficiently justified. Lemma 4.1 provides a one-sided correction, which does not bound the absolute smoothing bias.Moreover, the variance term in Eq. (15) should be estimated by

1n2∑i=1nΨ^i2,\frac{1}{n^2}\sum_{i=1}^{n}\widehat{\Psi}_i^2,

rather than

1n∑i=1nΨ^i2.\frac{1}{n}\sum_{i=1}^{n}\widehat{\Psi}_i^2. 2. Optimality claims. Efficiency for a fixed smoothed target does not establish optimal estimation of the original bounds, and the proposed smoothing selection procedure lacks a corresponding optimality result.Real-world interpretation. For Experiment 1, Table 1 reports a risk bound of 0.370.37, while the text states 0.250.25. Neither value alone establishes that launching is safe without an explicit risk tolerance.

Questions for authors
  1. How does Appendix C handle the Bernoulli counterexample above? Would correcting the treatment of probability mass points change the reported discrete-outcome results?

  2. Can the authors justify using b² as a squared-bias proxy and clarify the variance scaling in Eq. (16)? Since the experiments use the Lepski-style rule, how does its ordering by b address the same one-sided-bias limitation?

Reviewer_rkVJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the problem of estimating the distribution of treatment effects in observational studies or A/B testing. However, the treatment effect distribution is only partially identified in these settings. The resulting bounds on the treatment effect distribution are known as Makarov bounds in the literature. Makarov bounds can be efficiently estimated under either stylized assumptions on the outcome distribution or a margin assumption. However, the margin assumption fails when the treatment effect is constant for a subpopulation, a scenario that is particularly relevant in practice. This paper aims to develop efficient estimators of Makarov bounds without imposing the margin assumption. The main contributions are to (a) propose smoothed surrogates for the Makarov bounds, (b) derive semiparametrically efficient estimators of these surrogates, and (c) develop a procedure for selecting the smoothing parameters. Finally, numerical studies demonstrate that the proposed method outperforms existing methods in terms of mean squared error (MSE). The proposed method is also applied to real-world A/B testing data.

Critical strengths

Estimating the distribution of treatment effects in A/B testing is a practically important problem. A key strength of the paper is the development of an estimator based on smoothed surrogates designed for settings where the margin assumptions do not hold. The proposed estimator also yields favorable empirical results compared with existing methods under the settings mentioned above.

Critical weaknesses
  1. One limitation of Corollary 4.3 is that it does not account for the dependence of the smoothing parameters t1t_1 and t2t_2 on the sample size nn. The proposed procedure selects t1t_1 and t2t_2 using a data-driven approach, and, as far as I understand from Equation (16) (after correcting the factor to 1/n21/n^2; see my Q2), the optimal values of t1t_1 and t2t_2 should converge to infinity as n→∞n \to \infty. Thus, it would be important to extend the theoretical analysis to account for this dependence on nn, as the current result may not fully reflect the practical implementation of the proposed method.

  2. There is a lack of theoretical support for the empirical finding that their proposed method has a lower mean squared error (MSE) than the existing envelope estimator.

Questions for authors
  1. Based on the context and positioning of the citation in the Introduction, could the authors please verify whether “Kallus et al. (2022)” should instead refer to the single-author paper by Kallus (2022), What’s the Harm? Sharp Bounds on the Fraction Negatively Affected by Treatment?

  2. In equation 16, should it not be 1n2\frac{1}{n^2} instead of 1n\frac{1}{n}?

  3. In the empirical analysis, the right panel of Figure 2 (OHIE data, marginal one) does not clearly show whether the smoothing method outperforms the envelope method. Perhaps zooming in on the plot would help make the difference more visible.

  4. What is biased-corrected estimate of Makarov bounds in Figure 3? As far as, I understand, it is not defined in the paper.

  5. In Figure 3, the authors mention that the confidence intervals are displayed as shaded regions. However, these shaded regions are not clearly visible in any of the three experiments. Could the authors improve the visualization, perhaps by increasing the shading opacity or providing zoomed-in plots, to make the confidence intervals more distinguishable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.