acceptodds
Withdrawn from ICLR 2027

SMART: Sequential Monitoring with Adaptive Risk Testing for Dynamic A/B Experiments

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Online experiment design, also known as A/B testing, is central to modern mobile health, yet existing work mainly compares policies through average treatment effects. Mean-based testing can overlook clinically important tail risks, particularly when outcomes are skewed, heavy-tailed, or safety-critical. Recent work on distributional treatment effects (DTE) highlights the value of evaluating policy impacts beyond the mean, offering a full characterization of policy effects and greater robustness to heavy-tailed outcomes. However, existing methods are largely designed for offline data or randomized experiments and do not directly address real-time monitoring of temporally dependent data streams. In this paper, we propose SMART, a scalable and interpretable framework for real-time A/B testing with adaptive risk in dynamic healthcare settings. SMART builds on a linear-quadratic dynamic model to characterize policy-induced return distributions over time while accounting for carryover effects. We establish the asymptotic joint normality of the test statistics, which underpins valid DTE inference, and introduce a sequential monitoring procedure with a bootstrap-based stopping criterion for efficient online experimentation. Experiments on synthetic data and an OhioT1DM-based evaluation show that SMART can detect policy differences that are not captured by mean-based testing.

Primary area:
reinforcement learning
Keywords:
A/B Testing, Policy Evaluation, Distributional Treatment Effect

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9DcdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is focusing on the distributional treatment effects (DTE) for evaluating the policy impacts under real-time monitoring. This paper proposed SMART which us linear-quadratic dynamic model as backbone to estimate the distributions over time with asymptotic normality and valid inference.

Critical strengths
  1. The question of paper discussing is novel and practical in real-world application.
  2. paper is well-written, and the experiments includes synthetic and real-world application.
Critical weaknesses
  1. In the related work section, there is missing reference of Causal distributional estimation, e.g., https://arxiv.org/abs/2209.06203. And the section is badly written, since the author just dumped all the citations in without carefully distinguishing the key contribution of the papers. It makes the contribution of SMART weak.
  2. In the problem setup section, the paper never define the population of random variables. But the paper is focusing on the distributional effect. IMHO, here should have more clear definition and then the target estimand could be well-defined.
  3. Also there is missing the classic causal assumption which is need for ATE estimations. I think they are implicitly stated in the Linear dynamical system, but writing out would be easier for reader to read.
  4. Equation (2) and Algorithm 1 appear to use an uncentered multiplier estimator, whereas Appendix B.7 proves validity for a centered residual bootstrap around the original OLS estimate. Please clarify which version is implemented and whether δ^(q)\widehat{\delta}^{(q)} in Equation (5) is a complete bootstrap estimate or a centered perturbation, since this distinction directly affects the claimed sequential Type-I error control.

Minor:

  1. Line 045 there is a missing space between "mean outcomesJohari", and line 052 "discussedChernozhukov".
  2. Line 092, there are double brackets in the first sentence of related work section. "((see Larsen et al., 2024; Quin et al., 2024, for recent reviews)),"
Questions for authors

Questions for authors

  1. For the discrete-time linear dynamical system, the xt+1x_{t+1} is only rely on xtx_t and utu_t. Does this time-dependent enough? Or should the author consider more steps before time tt, e.g., xt−kx_{t-k} and ut−ku_{t-k}.
  2. I think in the problem setting section, the definition of ut=Kxtu_t = K x_t and the policy takes form π(x)=Kx\pi(x) = Kx are a bit contradict to each other. Is the ut∼Bern(π(x),1)u_t \sim \text{Bern}(\pi(x), 1) or what is the relationship between these two.
Reviewer_guAcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SMART, a framework for sequential comparison of risk-sensitive policy outcomes in dynamic A/B experiments. It combines a linear-quadratic model, return-distribution estimation, and bootstrap-calibrated monitoring boundaries to test differences in means, quantiles, and CVaR. The evaluation includes synthetic experiments, an OhioT1DM-based application, and robustness and design-sensitivity analyses.

Critical strengths
  1. The paper addresses a practically relevant problem by combining temporal dependence, tail-risk evaluation, and early stopping in dynamic A/B testing.

  2. The framework accommodates several risk measures and explicitly attempts to propagate transition-model estimation uncertainty into sequential inference.The empirical study includes 500-repetition main experiments and additional checks of model misspecification, patient heterogeneity, preprocessing, and monitoring schedules. Reporting failure cases is valuable.

Critical weaknesses
  1. The bootstrap algorithm and its proof are inconsistent. Equation (2) and Algorithm 1 apply mean-zero multipliers directly to the raw regression cross-products, producing parameter draws centered at zero before projection. Appendix B.7 instead analyzes weighted least squares with weights of mean one, while B.8 asserts a residual-based centered expansion without a valid derivation. The stated proof therefore does not establish validity of the implemented procedure as described.The empirical evidence does not adequately support the claimed sequential error control. Appendix C.2.3 reports null rejection rates of 0.09 under the default design and 0.19–0.20 under alternative spending schedules. Zero rejection under an interior null in the main experiments does not establish calibration at the null boundary. More precise boundary-null evaluations are needed.

  2. The experimental state construction conflicts with the stated stability assumption. Appendix C.1 includes a constant state coordinate that remains one, implying a unit eigenvalue in the full closed-loop matrix, whereas C1 requires strict stability. The treatment of this coordinate and its compatibility with the projection step need clarification.The clinical conclusions exceed the validation provided. The OhioT1DM evaluation uses six patients and artificially maps discrete insulin actions to continuous values. The relationship between observed data, simulated trajectories, and repeated evaluations is insufficiently clear, and the reported preprocessing sensitivity further limits claims of demonstrated clinical utility.

Questions for authors
  1. Which bootstrap construction was actually implemented: Eq. (2), the weighted least-squares procedure in B.7, or a centered residual-score procedure? Please reconcile the algorithm and proof and clarify whether correcting them changes the results.

  2. What nominal error level was used in the supplementary studies? Can the authors provide boundary-null calibration with more repetitions and explain the inflated rejection rates under alternative spending schedules?

  3. How is the constant state coordinate handled in the stability assumption and spectral projection? Which parts of the OhioT1DM evaluation use observed transitions, and which use simulated data?

Reviewer_8tdpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a framework for sequentially comparing risk-sensitive functionals of long-run policy returns in dynamic A/B testing with carryover effects. The proposed method estimates transition dynamics using OLS, approximates the innovation distribution using empirical residuals, and uses an existing return representation to estimate contrasts in means, quantiles, and CVaR. The method also combines multiplier-bootstrap inference with alpha-spending boundaries for early stopping at prespecified monitoring times.

Critical strengths
  • The problem addressed in the paper is practically relevant, especially when policies with similar average outcomes have different risk profiles, and carryover makes such comparisons more challenging than traditional A/B testing.
  • The proposed methodology is structurally clear and coherent.
Critical weaknesses
  • There're technical mistakes in the bootstrap construction, and the bootstrap procedure is inconsistent with its theoretical justification. Eq (2) and Algorithm 1 apply mean zero Gaussian multipliers to the raw cross-products and use the resulting coefficients directly as perturbed transition matrices. Before projection, these coefficients have conditional expectation zero, rather than being perturbations around the fitted dynamics. But the proof in the appendix analyzes weighted least squares, which is a different procedure.
  • There is another issue concerning uncertainty in the estimated innovation distribution. The DTE estimator depends on both the transition coefficients and the empirical residual law, but the bootstrap does not clearly reproduce the latter’s sampling uncertainty. The proof asserts a complete influence function expansion without showing that the algorithm reproduces all its components. These issues directly affect the variance estimator and sequential boundaries, so the validity claim can not established as written.
  • The novelty seems to be overstated relative to existing work, e.g., the stochastic return representation and CVaR-based distributional LQR evaluation come from Wang et al. (2023), and dynamic A/B testing with carryover, joint sequential asymptotics, multiplier-bootstrap calibration, and alpha spending are developed in Shi et al. (2023). Also, sequential inference for distributional targets already exists.
  1. Shi et al. (2023), “Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework.”
  2. Wang et al. (2023), “Policy Evaluation in Distributional LQR.”
Questions for authors
  • Which bootstrap was actually used in the experiments? the mean-zero, raw-cross-product construction in Eq (2) and Algorithm 1, or the weighted least-squares procedure analyzed in Appendix?
  • How does the bootstrap capture the sampling uncertainty of the empirical innovation law, beyond Monte Carlo simulation of returns conditional on that law? Please provide the relevant influence function decomposition or other arguments showing that the implemented bootstrap jointly reproduces uncertainty from transition estimation and innovation-law estimation.
  • What numerical nominal alpha, exact spending functions, and inner return-simulation and outer multiplier-bootstrap sample sizes were used?
  • Could the authors clarify the contributions and novelty or restate the novelty claim?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.