- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies continuous experimentation in which treatment arms can be introduced and removed based on previously observed data. It constructs treatment specific test using transformed Horvitz–Thompson scores and combines them with asynchronous e-LOND to control the online false discovery rate.
Critical strengths
- The framework combines evolving treatment sets, shared controls, continuous monitoring, and asynchronous multiple testing. The paper provides a coherent statistical formulation and an implementable approach to maintaining validity under predictable treatment changes.
- The main-text analysis establishes individual anytime-valid testing and online FDR control, and goes beyond validity by analyzing the asymptotic growth of the resulting evidence processes.
Critical weaknesses
- The methodological novelty seems to be marginal. Algorithm 1 adapts asynchronous e-LOND from Xu and Ramdas (2024). The Horvitz–Thompson score resemble the causal testing approach in Sandoval et al. (2026). The log-optimality argument builds on the portfolio regret of Waudby-Smith et al. (2025). The paper does not provide substantive new insights beyond combining these components.
- The significance of the optimality result and the efficiency benefit of restarting are not sufficiently demonstrated. For example, Theorem 4.5 does not characterize the finite-sample cost of repeated restarting in experiments with frequent treatment changes. Figure 2 does not provide enough information to explain when the reported restart advantage arises, and the power and sample-use experiments do not isolate that advantage against a no-restart version or another valid anytime procedure with matched monitoring opportunities.
- I also didn't see any code files submitted with the paper.
Questions for authors
- Could the authors provide more dicussion on the novelty? Please distinguish the new result from the adaptation of existing components to the experimental setting.
- What allocation probabilities, sub-experiment lengths, and treatment-change mechanism generate Figure 2(c)? How does the relative probability change across stages?
- some history indicators around Eq (14) use the current history , although predictable betting appears to require the preceding history . Please clarify the intended definitions and whether the proofs use these corrected versions.