acceptodds
Withdrawn from ICLR 2027

Always-On Experiments

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generative AI has dramatically accelerated the rate at which new treatments—from novel pharmaceuticals to online marketing campaigns—can be conceived and deployed. As a result, modern experimentation platforms often run continuously, with treatments added as they are ready and removed when they underperform. We formalize this "Always-On" experimental setting, in which treatments can be dynamically generated, added to, and removed from a running experiment, and study the statistical problem of deciding whether to accept or reject each treatment while controlling for the false discovery rate. We develop sequential tests that achieve time-uniform Type-I error control under arbitrary stopping times and "predictable" treatment schedules. Our approach builds on the testing-by-betting framework: we construct test supermartingales for testing the average treatment effect of each treatment, and show that the construction of these test supermartingales is growth-rate optimal in an almost-sure sense.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
randomized experiments, sequential hypothesis testing, false discovery rate

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_3hZQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continuous experimentation in which treatment arms can be introduced and removed based on previously observed data. It constructs treatment specific test using transformed Horvitz–Thompson scores and combines them with asynchronous e-LOND to control the online false discovery rate.

Critical strengths
  • The framework combines evolving treatment sets, shared controls, continuous monitoring, and asynchronous multiple testing. The paper provides a coherent statistical formulation and an implementable approach to maintaining validity under predictable treatment changes.
  • The main-text analysis establishes individual anytime-valid testing and online FDR control, and goes beyond validity by analyzing the asymptotic growth of the resulting evidence processes.
Critical weaknesses
  • The methodological novelty seems to be marginal. Algorithm 1 adapts asynchronous e-LOND from Xu and Ramdas (2024). The Horvitz–Thompson score resemble the causal testing approach in Sandoval et al. (2026). The log-optimality argument builds on the portfolio regret of Waudby-Smith et al. (2025). The paper does not provide substantive new insights beyond combining these components.
  • The significance of the optimality result and the efficiency benefit of restarting are not sufficiently demonstrated. For example, Theorem 4.5 does not characterize the finite-sample cost of repeated restarting in experiments with frequent treatment changes. Figure 2 does not provide enough information to explain when the reported restart advantage arises, and the power and sample-use experiments do not isolate that advantage against a no-restart version or another valid anytime procedure with matched monitoring opportunities.
  • I also didn't see any code files submitted with the paper.
Questions for authors
  • Could the authors provide more dicussion on the novelty? Please distinguish the new result from the adaptation of existing components to the experimental setting.
  • What allocation probabilities, sub-experiment lengths, and treatment-change mechanism generate Figure 2(c)? How does the relative probability πs(a)/[πs(a)+πs(0)])\pi_s(a)/[\pi_s(a)+\pi_s(0)]) change across stages?
  • some history indicators around Eq (14) use the current history Hi,sH_{i,s}, although predictable betting appears to require the preceding history Hi,s−1H_{i,s-1}. Please clarify the intended definitions and whether the proofs use these corrected versions.
Reviewer_ki9EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper considers randomised experiments with a shared control and treatments that can enter or leave over time. For each treatment it retains observations assigned to that treatment or control, normalises the assignment probabilities, and constructs a betting process from a transformed Horvitz–Thompson estimate. Asynchronous e-LOND allocates testing levels and controls FDR across discoveries. The paper proves asymptotic log-growth optimality within its specified betting class when a finite treatment schedule is fixed and all sub-experiment sizes grow. An appendix extends the construction to repeated units and finite treatment/outcome histories.

Critical strengths
  • The experimental problem is useful, and the treatment/control retention construction gives a clean way to use common controls while the active treatment set changes.
  • The main fresh-unit validity argument is sound with the intended nonnegative, bounded bets. Dependence among tests from shared controls is handled appropriately by an existing dependence-robust online FDR procedure. The proof does not require independent test statistics.
Critical weaknesses
  1. The repeated-unit validity result is false under the assumptions as written. Assumption B.2 requires the history-Markov property for active treatments a in A_s, but excludes the control. Lemma B.3 and its proof in D.6 nevertheless use that property for the control contrast. Consider three stages with active treatments c, b, a, independent fair treatment/control assignments, history length one, outcomes in {0,1,2}, and iid latent U~Bernoulli(1/2). At stage 1 both outcomes are 2U, revealing U; at stage 2 both outcomes are 1; at stage 3 treatment a returns 1 and control returns 2U after b, or 1 after control. All active-treatment outcomes satisfy B.2. Both stage-3 last-history effects are zero, but the full past reveals U. Betting only for units with U=0 and previous treatment b gives the paper's wealth factor 3/2 with probability 1/8 and one otherwise, so its expectation is 17/16>1. This refutes B.3/B.4 even after correcting the history indices to mean pre-treatment history. For this allowed predictable bettor, crossing probability tends to one under the null. UP's initial bet of 1/2 also has conditional expected factor 9/8 on the indicated full-history event, contradicting its claimed supermartingale property.

  2. Most components have close precedents, and the introductory novelty claims are overstated. Algorithm 1 is asynchronous e-LOND from Xu and Ramdas (2024), §5; listing that work among synchronous-only methods is incorrect. The transformed causal betting increment is algebraically identical to Sandoval et al. (2026), §6 after normalizing treatment/control propensities. The optimality architecture is already developed by Waudby-Smith et al. (2025), and history-specific portfolios have a direct precedent in Cover and Ordentlich (1996). Platform trials already permit treatment entry/exit, and anytime-valid A/B inference already permits continuous monitoring. The complete presented framework does not seem to have an exact predecessor but the paper needs to identify a substantive new obstacle beyond combining these components. The repeated-unit result would help distinguish the work if established correctly.

Questions for authors
  1. What precise result remains new after combining conditional causal betting, asynchronous e-LOND, and existing portfolio/numeraire theory? Please correct the characterization of Xu and Ramdas and compare the causal increment with Sandoval et al.
  2. Which γᵣ sequence was used, how should practitioners choose it for an unknown stream of treatments, and how sensitive are the reported power and enrollment results to this choice?
Reviewer_2SB5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies continuously monitored experiments in which treatments can be added and removed based on past observations. Using a potential outcomes formulation, the authors construct test statistics from inverse propensity weighted estimators and adapt an online multiple hypothesis testing method to control the false discovery rate. They also establish asymptotic log-optimality of the statistics constructed using Universal Portfolio within a specified comparator class for fixed finite treatment schedules.

Critical strengths

The paper combines the potential outcomes framework with online multiple hypothesis testing to construct statistically valid evidence for assessing treatment effects in continuously monitored experiments, where treatments can be dynamically added and removed. The authors also provide a detailed theoretical analysis of the proposed framework.

Critical weaknesses
  1. Insufficient explanation. The authors should make better use of figures, equations, and text, as the current presentation can be confusing. Several aspects need clearer explanation, including the meaning of “predictable” and how to interpret Figure 1, whose intended message is difficult to grasp from the caption alone. The theoretical presentation also places too much emphasis on successive mathematical derivations; more intuitive explanations are needed to convey the meaning behind the complex notation and results.

  2. Unclear contribution. Section 2 surveys relevant work, but the specific methodological and theoretical advances over existing online multiple testing and experimentation frameworks could be stated more clearly.

  3. Limited experimental results. The semi-synthetic evaluation uses a single data source. Additional application settings (or more constructed settings) and clearer discussion of the trade-offs with existing methods in the main text would strengthen the experimental evidence. The authors should also include comparisons with methods from the existing literature.

Questions for authors
  1. In Section 1 (Introduction, p. 1, lines 031–033), the authors state, “In this work we introduce Always-On Experimentation, a statistical design framework suited for the era in which the conception of treatments is no longer the bottleneck in the experimentation pipeline.” What evidence supports this broad claim?

  2. Section 3 defines sub-experiments as “those points in time in which treatments are added and removed from the running experiment.” What does “running experiment” mean in this context?

  3. The discussion following Theorem 4.5 identifies the random number of data points as a departure from prior work. What practical significance and additional technical challenges does this introduce?

  4. Assumption 3.3 allows data-dependent treatment additions and removals, whereas Theorem 4.5 fixes the number of sub-experiments and their active treatment sets. Does the log-optimality guarantee extend to adaptive treatment schedules, and under what conditions?

Reviewer_VXLjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I think this paper focus on one very real important problem,but the experiments are not very strong because there're lots of noise data in real world.

Critical strengths

The paper tries to solve a real difficulty, and gives a theoretical justification. and show a good result in experiments.

Critical weaknesses

The evidence for saving samples at equal power is not yet direct. It's not clear what happens to restart costs as the number of stages grows.

Questions for authors

The efficiency claims need more statistic results and evidence. and what will happen when the number of experiments grows with the total sample size?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.