SMART: Sequential Monitoring with Adaptive Risk Testing for Dynamic A/B Experiments

Online experiment design, also known as A/B testing, is central to modern mobile health, yet existing work mainly compares policies through average treatment effects. Mean-based testing can overlook clinically important tail risks, particularly when outcomes are skewed, heavy-tailed, or safety-critical. Recent work on distributional treatment effects (DTE) highlights the value of evaluating policy impacts beyond the mean, offering a full characterization of policy effects and greater robustness to heavy-tailed outcomes. However, existing methods are largely designed for offline data or randomized experiments and do not directly address real-time monitoring of temporally dependent data streams. In this paper, we propose SMART, a scalable and interpretable framework for real-time A/B testing with adaptive risk in dynamic healthcare settings. SMART builds on a linear-quadratic dynamic model to characterize policy-induced return distributions over time while accounting for carryover effects. We establish the asymptotic joint normality of the test statistics, which underpins valid DTE inference, and introduce a sequential monitoring procedure with a bootstrap-based stopping criterion for efficient online experimentation. Experiments on synthetic data and an OhioT1DM-based evaluation show that SMART can detect policy differences that are not captured by mean-based testing.